Email notifications have stopped being sent (resolved)
Added by Ladislav Kudláček 11 months ago
Hello, in our company we use the redmine system for about 100 people. From 11/15/2023, sending notifications when creating/updating a ticket stopped working. We have a backup from 1/11/2023 where everything works. Is it possible to find the differences in the settings please? We are not aware of any changes, so I would need to at least make a comparison.
thanks for ideas...
Replies (8)
RE: Email notifications have stopped being sent - Added by Ladislav Kudláček 11 months ago
I will add:
On a working server, the output from /var/log/maillog is:
Dec 6 14:12:53 helpdesk postfix/smtpd[3655]: connect from localhost.localdomain[127.0.0.1] Dec 6 14:12:53 helpdesk postfix/smtpd[3655]: 4F46048682: client=localhost.localdomain[127.0.0.1] Dec 6 14:12:53 helpdesk postfix/cleanup[3658]: 4F46048682: message-id=<redmine.journal-376785.20231206141252@mydomain.cz> Dec 6 14:12:53 helpdesk postfix/qmgr[2630]: 4F46048682: from=<helpdesk@mydomain.cz>, size=3233, nrcpt=17 (queue active) Dec 6 14:12:53 helpdesk postfix/smtpd[3655]: disconnect from localhost.localdomain[127.0.0.1]
...on a broken server:
Dec 6 14:13:06 helpdesk-clone postfix/smtpd[2889]: connect from localhost.localdomain[127.0.0.1] Dec 6 14:13:06 helpdesk-clone postfix/smtpd[2889]: warning: Illegal address syntax from localhost.localdomain[127.0.0.1] in RCPT command: <> Dec 6 14:13:06 helpdesk-clone postfix/smtpd[2889]: disconnect from localhost.localdomain[127.0.0.1]
RE: Email notifications have stopped being sent - Added by Mayama Takeshi 11 months ago
I would check if the version of postfix has changed:
$ postconf -d | grep mail_version mail_version = 3.5.6 milter_macro_v = $mail_name $mail_version
as a new version could have become stricter regarding the SMTP protocol.
If the version is the same in both servers, I would compare the RCPT in the SMTP messages sent by redmine to the server with ngrep:
$ ngrep -d any -q -W byline RCPT
RE: Email notifications have stopped being sent - Added by Ladislav Kudláček 11 months ago
The servers are very very old.
redmine version: 1.2.1
OS: CentOS 5
The ngrep tool is missing and won't install. It has more dependencies and CentOS is no longer supported.
I have the migration ticket here: https://www.redmine.org/boards/2/topics/69117 ... but I haven't received much advice yet :-)
The versions are the same.
A server where emails don't work:
[root@clone ~]# postconf -d | grep mail_version
mail_version = 2.3.3
milter_macro_v = $mail_name $mail_version
Server where emails works
[root@restore ~]# postconf -d | grep mail_version
mail_version = 2.3.3
milter_macro_v = $mail_name $mail_version
RE: Email notifications have stopped being sent - Added by Ladislav Kudláček 11 months ago
Ladislav Kudláček wrote in RE: Email notifications have stopped being sent:
The servers are very very old.
redmine version: 1.2.1
OS: CentOS 5The ngrep tool is missing and won't install. It has more dependencies and CentOS is no longer supported.
After solving this problem, we will migrate the data to the new operating system and the new redmine.
I have the migration ticket here: https://www.redmine.org/boards/2/topics/69117 ... but I haven't received much advice yet :-)
The versions postfix are the same.
A server where emails don't work:
[root@clone ~]# postconf -d | grep mail_version
mail_version = 2.3.3
milter_macro_v = $mail_name $mail_versionServer where emails works
[root@restore ~]# postconf -d | grep mail_version
mail_version = 2.3.3
milter_macro_v = $mail_name $mail_version
RE: Email notifications have stopped being sent - Added by Ladislav Kudláček 11 months ago
We tested:
On the server where e-mails are not being sent, we restored the database from the server where e-mails were still being sent (as of 1/11/2023). The result was that the mails started to be sent. As soon as we restored the database to this same server with current data, the mails stopped sending again...I think the problem will be somewhere in the database. Any ticket with bad ASCII encoding? I have no idea yet...
- echo "test" | mail -s "test" myname@mydomain.cz
Dec 7 02:17:42 helpdesk-clone postfix/pickup17313: D2B6D48730: uid=0 from=<root>
Dec 7 02:17:42 helpdesk-clone postfix/cleanup17767: D2B6D48730: message-id=<20231207011742.D2B6D48730@helpdesk-clone.mydomain.local>
Dec 7 02:17:42 helpdesk-clone postfix/qmgr2552: D2B6D48730: from=<root@helpdesk-clone.mydomain.local>, size=333, nrcpt=1 (queue active)
Dec 7 02:18:12 helpdesk-clone postfix/smtp17769: connect to gw.mydomain.cz[172.16.0.6]: Connection timed out (port 25)
Dec 7 02:18:12 helpdesk-clone postfix/smtp17769: D2B6D48730: to=<myname@mydomain.cz>, relay=none, delay=30, delays=0.02/0.01/30/0, dsn=4.4.1, status=deferred (connect to gw.mydomain.cz[172.16.0.6]: Connection timed out)
Postfix works. The problem will be somewhere in redmine system (smtp port is blocked by firewall, "Connection timed out" is correct)
RE: Email notifications have stopped being sent - Added by Mayama Takeshi 11 months ago
I would try to do a diff between the old and new server relevent folders like this:
diff -r old_redmine new_redmine
If this is generate a too large diff output, I would check just the config folder:
diff -r old_redmine/config/ new_redmine/config/
If nothing relevant seems to differ in redmine, I would also try to compare the postfix configuration:
diff -r /etc/postfix postfix_folder_from_the_other_server
Obs: do not post the results here unless you know there are no confidential information.
Regarding ngrep, you could capture the data using tcpdump:
tcpdump -i any -s 0 -w test.cap 'port 25'
and then read test.cap on a machine where ngrep is available.
RE: Email notifications have stopped being sent - Added by Ladislav Kudláček 11 months ago
I made exactly these distinctions. There really was a difference in the config, I adjusted it according to the working server, but it didn't help. I always edit confidential information (myname, mydomain etc...). So the output is modified. There is no difference in postfix, it is set identically... and according to the test from the command line, mail is sent via postfix. It doesn't just send from redmine.
RE: Email notifications have stopped being sent (resolved) - Added by Ladislav Kudláček 11 months ago
Ugh....resolved. The problem was in the mailbox of two users. There were hundreds of users. By method of elimination, I found that two users were responsible. Someone set mailbox usernames with accents.
Thanks very much Mayama Takeshi for reaction...