Project

General

Profile

Mass File Upload by mask from server

Added by Valeriy Zhuk 9 months ago

Hello everyone. In version 5.0.2, my script has stopped working, which used to add information about files to the database, allowing them to be downloaded later. It was a collection of daily builds of applications, where files were uploaded to the appropriate section based on specific "masks." Currently, only file information is being uploaded, but when attempting to download, a 404 error occurs. If files are attached manually, everything works fine. I am attaching the scripts. Please help. I'm sure that if it worked in older versions, it should work here as well.

Environment:
  Redmine version                5.0.2.stable
  Ruby version                   2.7.4-p191 (2021-07-07) [x86_64-linux-gnu]
  Rails version                  6.1.6
  Environment                    production
  Database adapter               Mysql2
  Mailer queue                   ActiveJob::QueueAdapters::AsyncAdapter
  Mailer delivery                sendmail
Redmine settings:
  Redmine theme                  Purplemine2-master (includes JavaScript)
SCM:
  Subversion                     1.14.1
  Git                            2.30.2
  Filesystem                     
Redmine plugins:
  no plugin installed

The main script is executed via a cron job, which in turn calls other scripts. I will simply refer to them as 1, 2, and 3 here.

1.sh

chmod -R g+rwx /home/devel/distr/*

echo Generating sql for distr...
cd /home/devel/distr/
echo '  deleting sql...'
/opt/redmine/1-update-distr-del-sql.sh > /opt/redmine/update-distr-del.sql
echo '  inserting sql...'
/opt/redmine/2-update-distr-ins-sql.sh > /opt/redmine/update-distr-ins.sql

echo Generating sql for nighlty distr...
cd /home/devel/distr/nightly
echo '  deleting sql...'
/opt/redmine/1-update-distr-del-sql.sh >> /opt/redmine/update-distr-del.sql
echo '  inserting sql...'
/opt/redmine/2-update-distr-ins-sql.sh >> /opt/redmine/update-distr-ins.sql

echo Exectuing deleting sql for distr...
/usr/bin/mysql --user=root --password=strondpass redmine < /opt/redmine/update-distr-del.sql

echo Exectuing inserting sql for distr...
/usr/bin/mysql --user=root --password=strongpass redmine < /opt/redmine/update-distr-ins.sql

echo Deleting older than 4 days nightly distr...
find /home/devel/distr/nightly/* -mtime +4 -exec rm {} \;

echo Done.

2.sh

for f in *
do
  stat -c '%y %n %s' $f | cut --complement -b '19-34' | awk '{print "\
    delete from attachments\
    where\
      container_type = \"Version\" and\
      container_id in (\
        select\
          v.id as v_id\
        from\
          versions v, projects p\
        where\
          v.project_id = p.id and\
          \"" $3 "\" regexp concat(\"^\", p.identifier, \"-\", v.name, \"[[:alnum:]*.period[:alnum:]*]*[[.hyphen.][:alnum:]*]*[.period](exe|deb|rpm)\")\
      )\
    ;\
  "}'
done

3.sh

for f in *
do
  if [ -f $f ]; then
#
# Replace this command with bellow to enable md5 checksum.
#
# stat -t '%Y-%m-%d %H:%M:%S' $f | /sbin/md5 -qp $f | awk 'BEGIN {RS="";FS=" |\n"} {print "\
#
# Also add '\"" $22 "\",\' after value "application/octet-stream".
#
  stat -c '%y %n %s' $f | cut --complement -b '19-34' | awk 'BEGIN {RS="";FS=" |\n"} {print "\
    insert into attachments (\
      container_id,\
      container_type,\
      filename,\
      disk_filename,\
      filesize,\
      content_type,\
      digest,\
      downloads,\
      author_id,\
      created_on,\
      description\
    ) values (\
      (\
        select\
          v.id as v_id\
        from\
          versions v, projects p\
        where\
          v.project_id = p.id and\
          \"" $3 "\" regexp concat(\"^\", p.identifier, \"-\", v.name, \"[[:alnum:]*.period[:alnum:]*]*[[.hyphen.][:alnum:]*]*[.period](exe|deb|rpm)\")\
      ),\
      \"Version\",\
      \"" $3 "\",\
      \"" ENVIRON["PWD"] "/" $3 "\",\
      " $4 ",\
      \"application/octet-stream\",\
      \"""\",\
      0,\
      3,\
      \"" $1 " " $2 "\",\
      \"" $3 "\"\
    );\
  "}'
  fi
done

Old distributions are being deleted first, and then new ones are uploaded. This used to work in older versions, but not anymore. You thought that maybe an additional field is required – you checked the database table, and there haven't been any significant changes here.

It's possible that additional "parameters" need to be specified for the files, but I've already hit a dead end.