Redmine

Hi All,

I have had a Redmine build running for over a year, and it's been sending emails happily without issue. The emails have recently begun to fail (on +/-08 MAR 21), and after some (too much, lol) time troubleshooting I'm at a loss for why. I've been on the phone several times with Google's tech support and they're stumped too. Any help or ideas would be appreciated!

My Redmine specs:

Environment
  Redmine version                3.3.1.stable
  Ruby version                   2.3.3-p222 (2016-11-21) [x86_64-linux-gnu]
  Rails version                  4.2.7.1

I am sending mail using Google's SMTP relay to send emails, with the following setup in REDMINE/config/configuration.yml:

email_delivery:
    delivery_method: :smtp
    smtp_settings:
      enable_starttls_auto: true
      address: "smtp-relay.gmail.com" 
      port: 587
      domain: "MY_DOMAIN.com" 
      authentication: :plain
      user_name: "MY_EMAIL@MY_DOMAIN.com" 
      password: "MY_APP_PASSWORD"

Note: Since I have 2FA enabled on my Google Workspace account, enabling "Less Secure Apps" is not possible. I am, however, using an App Password for the account, which is their suggested replacement for the capability.

When clicking the "Send a Test Email" link at the bottom of the Administration >> Settings page, I get the following error, in red, at the top of the page:

An error occurred while sending mail (535-5.7.8 Username and Password not accepted. Learn more at )

I see the following in /REDMINE/log/production.log:

Started POST "/admin/test_email" for 201.116.217.190 at 2021-05-25 23:28:19 +0000
 Processing by AdminController#test_email as HTML
   Parameters: {"authenticity_token"=>"__TOKEN__"}
   Current user: MY_USERNAME (id=1)
   Rendered mailer/test_email.text.erb within layouts/mailer (0.0ms)
   Rendered mailer/test_email.html.erb within layouts/mailer (0.1ms)
 Redirected to /settings?tab=notifications
 Completed 302 Found in 283ms (ActiveRecord: 3.7ms)
 Started GET "/settings?tab=notifications" for 201.116.217.190 at 2021-05-25 23:28:19 +0000
 Processing by SettingsController#index as HTML
   Parameters: {"tab"=>"notifications"}
   Current user: MY_USERNAME (id=1)
   Rendered settings/_general.html.erb (2.0ms)
   Rendered settings/_display.html.erb (4.0ms)
   Rendered settings/_authentication.html.erb (2.5ms)
   Rendered settings/_api.html.erb (0.5ms)
   Rendered settings/_projects.html.erb (3.5ms)
   Rendered queries/_columns.html.erb (3.6ms)
   Rendered settings/_issues.html.erb (9.1ms)
   Rendered settings/_attachments.html.erb (0.8ms)
   Rendered settings/_notifications.html.erb (2.1ms)
   Rendered settings/_mail_handler.html.erb (0.7ms)
   Rendered settings/_repositories.html.erb (4.6ms)
   Rendered common/_tabs.html.erb (32.6ms)
   Rendered settings/edit.html.erb within layouts/admin (32.8ms)
   Rendered admin/_menu.html.erb (1.6ms)
   Rendered layouts/base.html.erb (7.7ms)
 Completed 200 OK in 47ms (Views: 40.5ms | ActiveRecord: 3.3ms)

Now, here's the kicker: in speaking with Google's techs, they do not see any message send attempt on their side. So although the problem could certainly be somewhere on their side, it appears that for some reason the Redmine server is not sending out the message, but is still returning an error anyway.

Here's what makes it stranger:

• The error shown in Redmine is a Google error. This makes me think that Redmine is working fine.

• I have worked with the tech to try both authenticated access (as shown above) and IP whitelisted access (change the settings in the Google Workspace admin console, and comment out the auth/username/pwd fields in configuration.yml), and nothing has worked.

• Even stranger, according to the Google tech, if IP whitelisting is used, it should hand back (something like) an "Invalid IP Address" error and not a "Username and Password Not Accepted" error.

• To add insult to injury, this Redmine instance is installed on a Google Cloud Platform VM, so all of the comms are internal to Google's ecosystem.

• Google informs me that this can take 1-2 hours to propagate, but in most cases it's (effectively) immediate. In either case, we've been working this for some time and any propagation window should have expired by now. Plus, the tech can see my settings on his side, so they've clearly stored.

So... I'm at a loss. The Redmine logs don't show specifics on the email send process, so I can't get any more clarity there. The server is responding to the test email link, so I think that the UI side is working fine. And it's showing a Google-styled error, so I think that it's actually doing something and getting a response. And yet Google isn't seeing any calls on their side.

Has anyone experienced this before? Do you have any ideas to resolve it?

References:

• Yes, I've looked at Google's SMTP Relay article.

• Yes, I've looked at Redmine's Email Configuration page.

• Yes, I've restarted Redmine (alot), both using sudo touch restart.txt and restarting the entire VM.

Related questions:

• Is there any way to see inside the mail process? Maybe another log set that I haven't looked at, or a VERBOSE flag or something I could turn on?

• What's with the 302 redirect in the logs? I can't imagine that this is related to a mail problem, but I was surprised to see it and tbh I'm grasping at straws.

Thanks again...

Jeremy