Sporadic latency viewing mailboxes - Zimbra Forums (2024)

Ubuntu 16.04 LTS 64-bit
Zimbra 8.8.9_GA_3717.NETWORK

On Monday February 27th, I made changes to the Zimbra services in order to meet requirements for security and SSL testing. Changes involved zimbraReverseProxySSLProtocols, zimbraResponseHeader, zmlocalconfig.

After the changes were made, users began seeing increased latency when viewing mail, sending mail, and moving between mail folders. We closely monitored resources on the Zimbra server searching for issues with the proxies, nginx, and mailboxd. No issues have been seen. Monitoring with iostat, ntop, iotop, and nmon all looks normal.

Access to the user's mail will be fine for a few minutes and then have a 10 to 60 second latency. Oddly ** important** this issue is never seen when I open a user's mail from the admin interface. I am able to move through the folders without problem and send receive mail without problems or latency. Not having moved to at least 8.8.15, our paid support is void. Before we attempt to migrate to a new version of Zimbra, we would like to resolve this issue.

I do not know that the changes I made were the cause of the issues we are experiencing. I reverted to the previous configuration and restarted the services with no difference in the user experience.

Any ideas where to look are appreciated.

DAve

John Holder (of blessed memory) advised us that when Zimbra is really slow but CPU and other indicators show that the system isn't busy, then it's often one or more blocked Java threads and it's time to do some thread dumps.

You can always run (as root) /opt/zimbra/libexec/zmdiaglog -a -j -z to get a complete diagnostic dump, open a Support Case for your migration, report the specific changes that you have made to your 8.8.9 server in preparation for the migration, and ask them to provide an FTP site so you can upload the results of zmdiaglog.

Zimbra Support has a tool that automagically parses through the zmdiaglog output, and provides an analysis that often quickly identifies the root cause. Support can then quickly provide remediation.

I think we both know you likely may need to back out some of your changes, and going forward I would recommend making as few changes as possible to the existing 8.8.9 system until you get migrated to the new Zimbra system.

Hope that helps,
Mark

Thank you, I found the problem. I was using several blog/wiki posts as guidance to remove TLSv1 and TLSv1.1 to meet new security requirements. I did back the changes out Wednesday but found an issue thismorning. Clearly, my mistake, though my shell history does not show I made a typo.

zimbraReverseProxySSLProtocols in zmprov MUST match https.protocols and jdk.tls.client.protocols in zmlocalconfig exactly. If it does not, you get this message in the nginx.log and very slow connections.

2023/03/03 06:26:26 [warn] 9503#0: *884074 zmauth: an error occurs during zm lookup: timeout occurs when reading lookup response from handler, fall back to IPHASH to get the upstream route while SSL handshaking to lookup handler

Once I looked at the error, I looked up several other posts on it and then checked the /opt/zimbra/conf/localconfiog.xml and saw the problem. I ran zmlocalconfig again with the correct parameters and restarted all services. Latency is now gone, errors are now gone.

I have commitment from above that the Zimbra upgrade really needs to be above several other projects.

DAve