sheep

Most LDAP server software support some sort of replication between multiple servers. OpenLDAP in particular has support for the syncrepl overlay (link) and also has support for the less talkative delta sync replication.

The way LDAP replication works is through the tracking of the entryCSN and contextCSN operational attributes. if you wish to inspect these values you can run the following command:

ldapsearch -LLL -x -h ldap2 -D "dc=dashboarddemo1,dc=com" -b "dc=dashboarddemo1,dc=com" -w secret +

dn: dc=dashboarddemo1,dc=com
structuralObjectClass: domain
entryUUID: d21cb050-931b-1039-96db-232e8ebafa48
creatorsName: dc=dashboarddemo1,dc=com
createTimestamp: 20191104065524Z
entryCSN: 20191104065524.600162Z#000000#00c#000000
modifiersName: dc=dashboarddemo1,dc=com
modifyTimestamp: 20191104065524Z
contextCSN: 20191104065849.472236Z#000000#00c#000000
entryDN: dc=dashboarddemo1,dc=com
subschemaSubentry: cn=Subschema
auditContext: cn=accesslog
hasSubordinates: TRUE

dn: ou=People,dc=dashboarddemo1,dc=com
structuralObjectClass: organizationalUnit
entryUUID: d21cb51e-931b-1039-96dc-232e8ebafa48
creatorsName: dc=dashboarddemo1,dc=com
createTimestamp: 20191104065524Z
entryCSN: 20191104065524.600412Z#000000#00c#000000
modifiersName: dc=dashboarddemo1,dc=com
modifyTimestamp: 20191104065524Z
entryDN: ou=People,dc=dashboarddemo1,dc=com
subschemaSubentry: cn=Subschema
hasSubordinates: TRUE

The contextCSN attribute keeps track of the latest change to the database while the entryCSN contains the timestamp of when an entry was added. The CSNs (change sequence numbers) are assembled as follows:

TIMESTAMP#COUNT#SID#MOD

CSNs are delimited with the ‘#’ character and contain 4 fields. The first is the time stamp. In the entryCSN, the timestamp represents the time where a particular entry was added or updated. For the contextCSN, the timestamp will match the latest entryCSN’s timestamp. The 2nd entry is the counter, in case there are multiple CSNs for the same timestamp.The 3rd entry is the serverID. The last entry is the counter to order any modifications (not used in openLDAP).

Notice how the top naming context entry has both an entryCSN and a contextCSN, while the ou=People,dc=dashboarddemo1,dc=com entry only has an entryCSN. This is because the contextCSN is only tracked in the naming context entry.

Sometimes you will see multiple contextCSN entries:

dn: dc=example,dc=com
objectClass: top
objectClass: organization
objectClass: dcObject
o: example
dc: example
structuralObjectClass: organization
creatorsName: dc=example,dc=com
entryUUID: 694799e6-8d4a-1038-9545-4741ae48c364
createTimestamp: 20181206022850Z
description: The description
entryCSN: 20190601190445.636053Z#000000#002#000000
modifiersName: dc=example,dc=com
modifyTimestamp: 20190601190445Z
contextCSN: 20181211001059.281098Z#000000#000#000000
contextCSN: 20191016005854.929834Z#000000#002#000000
contextCSN: 20190725143455.348435Z#000000#003#000000
contextCSN: 20191102035617.854359Z#000000#004#000000
contextCSN: 20191027025837.395188Z#000000#00c#000000

This is because openLDAP stores a contextCSN per each producer it is replicating from. In the above case there are 4 other servers that the current producer is obtaining data from (note 000 does not count, this entry is generated automatically for the producer if a serverId entry is not specified. This can be removed through the slapadd -w flag).

So how can we ensure our servers have the latest data and are replicating properly? We compare all the contextCSNs between each server! More specifically we compare the difference in the contextCSN timestamps and match them to each serverID.

The excellent ltb project (link) has a nice plugin that does these all the calculations for us. Note that to get the script running you will need the Net::LDAP and Time::Piece perl modules installed as well as the mdb_stat utility installed (found in the lmdb package). After all the dependencies are installed, you can run the following command:

./check_ldap_syncrepl_status.pl -H ldap2 -w 1 -c 2 -D "dc=example,dc=com" -P secret -U ldap3 -I 001

Assuming you have a 2 way MMR (multi-master replication) setup, you would run the following 2 commands:

./check_ldap_syncrepl_status.pl -H ldap2 -w 1 -c 2 -D "dc=example,dc=com" -P secret -U ldap3 -I 001
./check_ldap_syncrepl_status.pl -H ldap2 -w 1 -c 2 -D "dc=example,dc=com" -P secret -U ldap3 -I 002

You should see the following if there are no issues:

OK - directories are in sync (W:1 - C:2)

The above commands compares the contextCSNs of both servers. Both contextCSN attributes with the matching serverID flag will get compared, if the difference in the timestamps differ by greater than the -w warning or -c critical threshold flags, the script will return an error. If your servers are perfectly in sync, you should never see large replication delays.

You can then add these commands as service checks into Nagios or your preferred monitoring solution. Now you will be able to know if any of your openLDAP servers are out of sync (having any kind significant replication delays).

© Copyright 2020 Rex Consulting, Inc. – All rights reserved