Most LDAP server software support some sort of replication between multiple servers. OpenLDAP in particular has support for the syncrepl overlay (link) and also has support for the less talkative delta sync replication.
The way LDAP replication works is through the tracking of the entryCSN and contextCSN operational attributes. if you wish to inspect these values you can run the following command:
ldapsearch -LLL -x -h ldap2 -D "dc=dashboarddemo1,dc=com" -b "dc=dashboarddemo1,dc=com" -w secret + dn: dc=dashboarddemo1,dc=com structuralObjectClass: domain entryUUID: d21cb050-931b-1039-96db-232e8ebafa48 creatorsName: dc=dashboarddemo1,dc=com createTimestamp: 20191104065524Z entryCSN: 20191104065524.600162Z#000000#00c#000000 modifiersName: dc=dashboarddemo1,dc=com modifyTimestamp: 20191104065524Z contextCSN: 20191104065849.472236Z#000000#00c#000000 entryDN: dc=dashboarddemo1,dc=com subschemaSubentry: cn=Subschema auditContext: cn=accesslog hasSubordinates: TRUE dn: ou=People,dc=dashboarddemo1,dc=com structuralObjectClass: organizationalUnit entryUUID: d21cb51e-931b-1039-96dc-232e8ebafa48 creatorsName: dc=dashboarddemo1,dc=com createTimestamp: 20191104065524Z entryCSN: 20191104065524.600412Z#000000#00c#000000 modifiersName: dc=dashboarddemo1,dc=com modifyTimestamp: 20191104065524Z entryDN: ou=People,dc=dashboarddemo1,dc=com subschemaSubentry: cn=Subschema hasSubordinates: TRUE
The contextCSN attribute keeps track of the latest change to the database while the entryCSN contains the timestamp of when an entry was added. The CSNs (change sequence numbers) are assembled as follows:
TIMESTAMP#COUNT#SID#MOD
CSNs are delimited with the ‘#’ character and contain 4 fields. The first is the time stamp. In the entryCSN, the timestamp represents the time where a particular entry was added or updated. For the contextCSN, the timestamp will match the latest entryCSN’s timestamp. The 2nd entry is the counter, in case there are multiple CSNs for the same timestamp.The 3rd entry is the serverID. The last entry is the counter to order any modifications (not used in openLDAP).
Notice how the top naming context entry has both an entryCSN and a contextCSN, while the ou=People,dc=dashboarddemo1,dc=com entry only has an entryCSN. This is because the contextCSN is only tracked in the naming context entry.
Sometimes you will see multiple contextCSN entries:
dn: dc=example,dc=com objectClass: top objectClass: organization objectClass: dcObject o: example dc: example structuralObjectClass: organization creatorsName: dc=example,dc=com entryUUID: 694799e6-8d4a-1038-9545-4741ae48c364 createTimestamp: 20181206022850Z description: The description entryCSN: 20190601190445.636053Z#000000#002#000000 modifiersName: dc=example,dc=com modifyTimestamp: 20190601190445Z contextCSN: 20181211001059.281098Z#000000#000#000000 contextCSN: 20191016005854.929834Z#000000#002#000000 contextCSN: 20190725143455.348435Z#000000#003#000000 contextCSN: 20191102035617.854359Z#000000#004#000000 contextCSN: 20191027025837.395188Z#000000#00c#000000
This is because openLDAP stores a contextCSN per each producer it is replicating from. In the above case there are 4 other servers that the current producer is obtaining data from (note 000 does not count, this entry is generated automatically for the producer if a serverId entry is not specified. This can be removed through the slapadd -w flag).
So how can we ensure our servers have the latest data and are replicating properly? We compare all the contextCSNs between each server! More specifically we compare the difference in the contextCSN timestamps and match them to each serverID.
The excellent ltb project (link) has a nice plugin that does these all the calculations for us. Note that to get the script running you will need the Net::LDAP and Time::Piece perl modules installed as well as the mdb_stat utility installed (found in the lmdb package). After all the dependencies are installed, you can run the following command:
./check_ldap_syncrepl_status.pl -H ldap2 -w 1 -c 2 -D "dc=example,dc=com" -P secret -U ldap3 -I 001
Assuming you have a 2 way MMR (multi-master replication) setup, you would run the following 2 commands:
./check_ldap_syncrepl_status.pl -H ldap2 -w 1 -c 2 -D "dc=example,dc=com" -P secret -U ldap3 -I 001 ./check_ldap_syncrepl_status.pl -H ldap2 -w 1 -c 2 -D "dc=example,dc=com" -P secret -U ldap3 -I 002
You should see the following if there are no issues:
OK - directories are in sync (W:1 - C:2)
The above commands compares the contextCSNs of both servers. Both contextCSN attributes with the matching serverID flag will get compared, if the difference in the timestamps differ by greater than the -w warning or -c critical threshold flags, the script will return an error. If your servers are perfectly in sync, you should never see large replication delays.
You can then add these commands as service checks into Nagios or your preferred monitoring solution. Now you will be able to know if any of your openLDAP servers are out of sync (having any kind significant replication delays).
© Copyright 2020 Rex Consulting, Inc. – All rights reserved
