Redmine

Hello Redmine community,

I'd like to share a complex issue we diagnosed to help anyone who might face a similar situation.

1. Environment
   - OS: Ubuntu Server 22.04 LTS
   - Database: PostgreSQL 14.11
   - Server: Apache 2.4.52 + Phusion Passenger 6.0.20
   - Virtualization: VMware

1. Symptoms
   - Data inconsistency: Issue IDs appeared and disappeared (e.g., MAX alternating between 989 and 1023).
   - Session issues: Users were randomly logged out.
   - SSH connection issues: SSH sessions frequently disconnected with "client_loop: send disconnect: Broken pipe".
   - 500 errors: The application returned "500 Internal Server Error" intermittently.
   - Clock drift: `systemd-timesyncd` reported an extreme clock drift of -12,922 ppm.

1. Root Cause
   The root cause was two different virtual machines configured with the same IP address (192.168.2.164). Both VMs were running Redmine but were in different states. The network switch alternated traffic between the two, causing all the erratic behavior. The two VMs had:

- Different `boot_id`: The kernel's boot ID changed between SSH connections, which is impossible on a single, continuously running machine.
- Different uptimes: The `uptime -s` command showed different system start times.

1. How We Diagnosed It
   1. We monitored the kernel's `boot_id` from a separate machine using a simple script. We saw two different boot IDs alternating in the log.
   2. We confirmed with `uptime -s` that the two systems had started at completely different times.
   3. We ruled out a simple IP conflict by verifying that both machines had different MAC addresses in the ARP table.
   4. The final piece of evidence was that one VM had our SSH key and the other didn't, causing "Permission denied" errors when the traffic was routed to the second VM.

1. Solution
   The final solution required the infrastructure team to:
   - Identify the two VMs.
   - Shut down the duplicate VM.
   - Ensure only the correct VM is running with that IP address.

This experience highlights the importance of checking for duplicate IPs and instances when facing mysterious, intermittent issues with data and application stability.

I hope this helps someone else in the future.

Best regards,
Jazmin