Troubleshooting

This page describes common issues which may occur during day-to-day operations. As a rule before contacting Support you need to collect logs and other diagnostics information which may be relevant to the occurred issue. Also in case you have deployment in the not officially supported environment you may be required to reproduce issue in the supported environment.

Logs and diagnostic data collection

The following data may be required to be collected and provided while performing troubleshooting:

  1. elDoc server logs: stored in the ../elDoc/logs directory
  2. elDoc application logs: accessible via Administration → System logs
    • Note: Log records with level [error] may contain detailed technical information about the error. Hover mouse cursor over log record date or level type and copy the detailed error stacktrace.
  3. 3rd components (MongoDB, Solr, elDoc-jmc) logs: stored in the respective folders according to the component's documentation
  4. OS level logs: stored according to the respective OS documentation

Know issues troubleshooting

This area will be extended with known issues and technics on troubleshooting the common issues.

1) Out-Of-Memory related issues

Out-Of-Memory issues are usually related to the IDP module and may occur due to not optimal configuration of the Recognition Rules, what creates high-load on the system as IDP phase is not optimized.

Recommendations are the following:

  1. Review and optimize recognition forms by adjusting keywords/minus-keywords;
  2. Suspend unused recognition forms;
  3. Review and adjust elDoc server system service configuration in terms of increasing allowed memory (-Xmx) and adjusting number of parallel threads for the IDP sub-system and whole elDoc server.

2) elDoc system service not starting after server reboot

In some circumstances you may found out that after server reboot elDoc system service (or other related dependency services) not starting automatically.

Issuing the following command "journalctl -u eldoc.service" or "journalctl -b" gives the output similar to:

systemd[1]: Cannot add dependency job for unit eldoc.service, ignoring: Unit not found.

After rebooting, performing a "systemctl start eldoc" or "systemctl status eldoc" command gives the following output:

# systemctl status eldoc.service
Unit eldoc.service could not be found.

The issue is very likely due to the service unit file, usually in /etc/systemd/system , being a symbolic link to a non-root filesystem.

In the example below, eldoc.service file is a symbolic link to the service file in /local/elDoc:

# ls -l /etc/systemd/system/eldoc.service
[...] eldoc.service -> /local/elDoc/eldoc.service

# grep "/local" /etc/fstab
/dev/mapper/datavg /local xfs defaults 0 0

Solution

  • Disable service and remove the symbolic links (service unit file and links created to enable it)
  • Copy the eldoc.service file to /etc/systemd/system and reload systemd by executing "systemctl daemon-reload"
  • Add a drop-in to instruct systemd to start the service after mounting /local
# systemctl edit /etc/systemd/system/eldoc.service
>>> opens an editor
[Unit]
RequiresMountsFor=/local
>>> save and close the editor
  • It is recommended to add the same drop-in into all custom service files created during elDoc system deployment
  • After completing editing service files perform systemd reload and enable service to start automatically
systemctl daemon-reload
systemctl enable eldoc.service

Taking the steps described above will help to resolve described issue.

3) Active Directory automatic synchronization fails

At some point administrators may find-out that automatic users directory synchronization with Active Directory can not be performed and the following log records appear in the system log:

[2022-02-03T17:09:56.041+0200] [] [SEVERE] [] [ua.eldoc.beans.LdapBean] [tid: _ThreadID=109 _ThreadName=http-thread-pool::http-listener(1)] [timeMillis: 1641452966013] [levelValue: 1000] [[
  LDAP sync failed, due to not recoverable error...
org.adsync4j.api.InvocationIdMismatchException: Expected Invocation ID is 1933a25f-29f9-5889-b1e5-4858292115e1, but retrieved ID is 9e0761a4-3581-9e73-ab7f-91c0c3e65932

An Invocation ID is an identification number that uniquely identifies a database within AD. Invocation IDs change during the restore process to make sure replication is consistent. It will only change when an Active Directory restore happens. In a restore process, it will change; the otherwise, the existing domain controllers will identify it as an existing domain controller and will not replicate the data over.

Log message shown above could be caused by change of the Invocation ID on the Active Directory side due to rollback operation or simply by changing the target server used for replication for example when using a load balancing setup (multiple LDAP Servers) in a round robin configuration. To avoid such issues it is recommended to point elDoc to the specific AD-server which doesn't change randomly and take care about rollback procedures on the AD-side, and in case any performed - revisit elDoc Configuration page and manually perform full-sync with Active Directory.

Performing full-sync with Active Directory manually will store the updated Invocation ID and USN-number, as such all other scheduled synchronizations with AD will work fine.

As another solution you may try to setup a connect to the Global Catalog (GC) port 3268 (example, ldap://test.domain.com:3268) instead of using the standard LDAP port 389 (example, ldap://test.domain.com:389) if the Domain Controller is also the Global Catalog server.