Using Amazon S3 for storage

Added by Christopher Dell about 5 years ago

Hello,

I'm trying to find some documentation about adding S3 support to Redmine be it by hacking the codebase or via a plugin. My current install is hosted on Heroku (which has a read only filesystem) hence the need for file uploads to be handled somewhere else in the cloud.

Has anyone been confronted with this issue before? Is there plugin in the works that I might be able to contribute to?
If not, do any of the devs know of any pitfalls that might be encountered before developing such a plugin? (ie, in principle should it simply be a matter of overloading the Attachment model?).

Any information would be much appreciated,

Chris

Replies (33)

RE: Using Amazon S3 for storage - Added by Christopher Dell about 5 years ago

After looking at the source code I'd be tempted to add in paperclip to the core - that would pave the way for a dead easy S3 implementation - any thoughts?

RE: Using Amazon S3 for storage - Added by Eric Davis about 5 years ago

Go right ahead, as long as you submit a patch for it ;)

I've been taking a look at paperclip for Redmine myself. I have one Redmine on Heroku (demo.redmine.org) but I'd like to be able to put my main Redmine's files on S3 to have more flexibility in hosting. I think the easiest path would be to create a Redmine plugin that overrode the core's Attachment model and saved to S3. That would make it easy to keep it compatible with the core Redmine until it's ready to be merged. If you could post it to Github, I might be able to help out once I finish up a few other projects.

Eric Davis

RE: Using Amazon S3 for storage - Added by Christopher Dell about 5 years ago

OK I'll have a go - not entirely sure about the best way of doing this but I guess it will become obvious as I plug away. Will post updates here.

RE: Using Amazon S3 for storage - Added by Shereef Bishay about 5 years ago

+1 for getting file uploads to work with S3. I too am hosted on heroku and would love to see this working. Christopher let me know how it's going, and if you need a hand. I'm no expert but will do what I can.

RE: Using Amazon S3 for storage - Added by Christopher Dell about 5 years ago

OK, so a first stab at an S3 plugin can be found here : http://github.com/tigrish/redmine_s3

I decided not implement paperclip in the end since it meant changing a whole bunch of the Attachment code.
Haven't had time yet far to tidy it up, add tests and sort out all of the plugin pages but it basically works.

Have tested on Heroku and it seems to be working fine. Yay!

More updates to come over the course of the next couple of days....

RE: Using Amazon S3 for storage - Added by Eric Davis about 5 years ago

Great, I'll try it on http://demo.redmine.org.

Eric Davis

RE: Using Amazon S3 for storage - Added by Andrzej Giniewicz about 5 years ago

That plugin looks great! I wasn't reading its code yet or testing it, I have question before I try - Is it designed to work with one S3 configuration per redmine instance or project? Also is it possible to configure S3 access data trough redmine administration options or it needs file manipulations?

RE: Using Amazon S3 for storage - Added by Christopher Dell about 5 years ago

@Andrzej - The plugin is made for 1 s3 bucket per redmine instance ie. all projects use the same S3 bucket.

All of the configuration is done with a yml file, so no it is not yet possible to configure this via the redmine administration area.

RE: Using Amazon S3 for storage - Added by Jim Mulholland almost 5 years ago

Nice job on the S3 plugin! Are there any other gotcha's to hosting Redmine on Heroku?

We are thinking about migrating over from Slicehost.

RE: Using Amazon S3 for storage - Added by Christopher Dell almost 5 years ago

@jim - the only thing that is problematic now is source control. I seem to remember seeing a github plugin but haven't really investigated it yet...

RE: Using Amazon S3 for storage - Added by Jim Mulholland almost 5 years ago

Thanks for the heads up on source control. I'll have to look into that before making the switch.

RE: Using Amazon S3 for storage - Added by Eric Davis almost 5 years ago

Jim Mulholland:

I'm running http://demo.redmine.org on Heroku. It's code is public on Github so feel free to steal what you need. A few gotchas:

  • Use heroku's config for storing your session store.
  • You will need to start Redmine locally and add your Redmine plugin's assets to git (public/plugin_assets). When Rails Engines starts, it copies those files but Heroku has a read-only filesystem so it will fail.
  • If you add SCM repositories, your database will grow really fast. I had demo.redmine.org up to about a 500MB database, well over Heroku's free plan.

Let me know if you need any help, I've deployed a few Redmine to Heroku and am planning to deploy a lot more of them there.

Eric Davis

RE: Using Amazon S3 for storage - Added by Shereef Bishay almost 5 years ago

Eric,

First off thanks for all the great dev work! Truly inspiring.

I've got redmine hosted on Heroku, and can't figure out how to get the repositories up and running. I'm esp. interesting in getting GIT working. I don't mind paying a little for the DB space.

Can you point me in the write direction? What am I not seeing?

thx in advance

RE: Using Amazon S3 for storage - Added by Alain D. almost 5 years ago

Thank you so much Christopher, it's working very well and I can at last store files on my Heroku hosted Redmine !

RE: Using Amazon S3 for storage - Added by Obiwan Chinobi almost 5 years ago

Eric Davis wrote:

  • You will need to start Redmine locally and add your Redmine plugin's assets to git (public/plugin_assets). When Rails Engines starts, it copies those files but Heroku has a read-only filesystem so it will fail.

Could you please elaborate on this? I setup rm to run locally, but how "add your Redmine plugin's assets to git (public/plugin_assets)"?

I am encountering the error when running rake db:migrate on heroku:

Read-only file system - /disk1/home/slugs/93246_e56aa89_a702/mnt/public/plugin_assets

RE: Using Amazon S3 for storage - Added by Christopher Dell almost 5 years ago

I seem to recall it being along the lines of :

# edit .gitignore - remove the /public/plugin_assets line
git add public/plugin_assets
git commit -a -m "Adding in the plugin_assets dir" 
git push heroku master

The idea is to make sure the directory is already created so that Redmine's initialization script doesn't try to create it on the read-only file system.

RE: Using Amazon S3 for storage - Added by Obiwan Chinobi almost 5 years ago

That worked. Thanks!

RE: Using Amazon S3 for storage - Added by Eric Davis almost 5 years ago

Christopher Dell:

Great job on the plugin. Installed it, configured my S3 account, and it worked perfectly the first time.

I've forked it and will try to add some more features. I saw you wanted a gemspec eh, I just happen to have a script for that. ;)

Eric Davis

RE: Using Amazon S3 for storage - Added by Michael Bilderback over 4 years ago

Eric, Chris,

Let me join the others in saying fabulous work on this code.

I was wondering if I could get some help though :) So, I would like to host my redmine on Heroku. I have the redmine_s3 plugin installed and running on my localhost just fine. It writes to the correct s3 bucket and all from the localhost instance. However, when I deploy to Heroku and try to upload files, it fails. Here is a snippet from the heroku logs for the app.

Processing ProjectsController#add_file (for 70.166.89.18 at 2010-01-29 11:37:30) [POST]
Parameters: {"commit"=>"Add", "action"=>"add_file", "authenticity_token"=>"CpduZCIaIZo4boOKr8RM4aF4yEQk21COR4ut9Z6LUzA=", "id"=>"big-bluff", "controller"=>"projects", "attachments"=>{"1"=>{"description"=>"trying again", "file"=>#<File:/tmp/RackMultipart.6075.1>}}}

Errno::ENOENT (No such file or directory - /disk1/home/slugs/122687_624f041_85c8/mnt/files/100129113730_S3-Browser-1.1.zip):
app/models/attachment.rb:79:in `initialize'
app/models/attachment.rb:79:in `open'
app/models/attachment.rb:79:in `before_save'
app/controllers/application_controller.rb:261:in `attach_files'
app/controllers/application_controller.rb:258:in `each_value'
app/controllers/application_controller.rb:258:in `attach_files'
app/controllers/projects_controller.rb:306:in `add_file'
/home/heroku_rack/lib/static_assets.rb:9:in `call'
/home/heroku_rack/lib/last_access.rb:25:in `call'
/home/heroku_rack/lib/date_header.rb:14:in `call'
thin (1.0.1) lib/thin/connection.rb:80:in `pre_process'
thin (1.0.1) lib/thin/connection.rb:78:in `catch'
thin (1.0.1) lib/thin/connection.rb:78:in `pre_process'
thin (1.0.1) lib/thin/connection.rb:57:in `process'
thin (1.0.1) lib/thin/connection.rb:42:in `receive_data'
eventmachine (0.12.6) lib/eventmachine.rb:240:in `run_machine'
eventmachine (0.12.6) lib/eventmachine.rb:240:in `run'
thin (1.0.1) lib/thin/backends/base.rb:57:in `start'
thin (1.0.1) lib/thin/server.rb:150:in `start'
thin (1.0.1) lib/thin/controllers/controller.rb:80:in `start'
thin (1.0.1) lib/thin/runner.rb:173:in `send'
thin (1.0.1) lib/thin/runner.rb:173:in `run_command'
thin (1.0.1) lib/thin/runner.rb:139:in `run!'
thin (1.0.1) bin/thin:6
/usr/local/bin/thin:20:in `load'
/usr/local/bin/thin:20

Rendering /disk1/home/slugs/122687_624f041_85c8/mnt/public/500.html (500 Internal Server Error)

Any idea of what I might have missed in the install or setup to cause this?

Thanks in advance, Mike

RE: Using Amazon S3 for storage - Added by Christopher Dell over 4 years ago

Hi Mike,

From what I can remember from installing Redmine on Heroku, you need to :

  1. Open up .gitignore and remove the line that says /files/*
  2. git add files (you'll probably want to make sure that there aren't any files in your local directory first)
  3. commit, and push to heroku

Let me know if that helps,

Chris

RE: Using Amazon S3 for storage - Added by Michael Bilderback over 4 years ago

Chris,

Thanks for the prompt reply. Your advice led me down the right path because once I push /files into heroku...I realized that the heroku install was still trying to write to the file system rather than s3. So I went to reconfiguring the gem bundle and got it working. Its now working just as designed.

Mike

RE: Using Amazon S3 for storage - Added by Barry Skidmore over 4 years ago

I have installed both the S3 plugin and the Google IFrame plugin.

They work locally just fine but when I push my git and db to heroku the plugins do not show up.??

I am making sure that I am running production on my local machine and pushing my production db url to heroku.

Anyone have a clue why this might be?

RE: Using Amazon S3 for storage - Added by Christopher Dell over 4 years ago

Hi Barry,

You need to make sure you're pushing your git repo to Heroku, not just the db.

Hope that helps,

Chris

RE: Using Amazon S3 for storage - Added by Igor Balk about 4 years ago

I'm trying to install S3 plugin to work with Redmine 9.2 and getting following error:

/var/spool/redmine/rm_0196/vendor/plugins/redmine_s3/lib/S3.rb:251:in `make_request': undefined method `empty?' for nil:NilClass (NoMethodError)
from /var/spool/redmine/rm_0196/vendor/plugins/redmine_s3/lib/S3.rb:157:in `create_bucket'
from /var/spool/redmine/rm_0196/vendor/plugins/redmine_s3/lib/redmine_s3/connection.rb:43:in `create_bucket'
from /var/spool/redmine/rm_0196/vendor/plugins/redmine_s3/rails/../init.rb:17
from /var/spool/redmine/rm_0196/vendor/rails/activesupport/lib/active_support/callbacks.rb:182:in `call'
from /var/spool/redmine/rm_0196/vendor/rails/activesupport/lib/active_support/callbacks.rb:182:in `evaluate_method'
from /var/spool/redmine/rm_0196/vendor/rails/activesupport/lib/active_support/callbacks.rb:166:in `call'
from /var/spool/redmine/rm_0196/vendor/rails/activesupport/lib/active_support/callbacks.rb:90:in `run'
from /var/spool/redmine/rm_0196/vendor/rails/activesupport/lib/active_support/callbacks.rb:90:in `each'
from /var/spool/redmine/rm_0196/vendor/rails/activesupport/lib/active_support/callbacks.rb:90:in `send'
from /var/spool/redmine/rm_0196/vendor/rails/activesupport/lib/active_support/callbacks.rb:90:in `run'
from /var/spool/redmine/rm_0196/vendor/rails/activesupport/lib/active_support/callbacks.rb:276:in `run_callbacks'
from /var/spool/redmine/rm_0196/vendor/rails/actionpack/lib/action_controller/dispatcher.rb:51:in `send'
from /var/spool/redmine/rm_0196/vendor/rails/actionpack/lib/action_controller/dispatcher.rb:51:in `run_prepare_callbacks'
from /var/spool/redmine/rm_0196/config/../vendor/rails/railties/lib/initializer.rb:631:in `prepare_dispatcher'
from /var/spool/redmine/rm_0196/config/../vendor/rails/railties/lib/initializer.rb:185:in `process'
from /var/spool/redmine/rm_0196/config/../vendor/rails/railties/lib/initializer.rb:113:in `send'
from /var/spool/redmine/rm_0196/config/../vendor/rails/railties/lib/initializer.rb:113:in `run'
from /var/spool/redmine/rm_0196/config/environment.rb:23
from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/rails.rb:147:in `rails'
from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:113:in `cloaker_'
from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:149:in `call'
from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:149:in `listener'
from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:99:in `cloaker_'
from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:50:in `call'
from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:50:in `initialize'
from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:84:in `new'
from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:84:in `run'
from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/command.rb:212:in `run'
from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:281
from /usr/local/bin/mongrel_rails:19:in `load'
from /usr/local/bin/mongrel_rails:19

Here is my s3.yml configuration

production:
access_key_id:XXXXXXXXXXXXXXXXXX
secret_access_key:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
bucket:bucket_v1
cname_bucket: false

development:
access_key_id:
secret_access_key:
bucket:
cname_bucket: false

RE: Using Amazon S3 for storage - Added by Daudi Amani about 4 years ago

Hi,
Am trying to get the redmine_s3 plugin working on heroku. It appears to be working on local dev machine and appears in admin/settings/plugins listing. It does not appear in the plugins listing when I'm on heroku. Mike said that he "went to reconfiguring the gem bundle and got it working." I'm new to redmine and do not see the documentation on how to reconfigure the gem bundle for a plugin like this. Do I need an entry in heroku's required .gems? Do I need a config.gem "redmine_s3" in my environment.rb file?

thanks
David

1 2 (1-25/33)