When a slave is many diffs behind, send these in batches of up to
50, then handle other slaves.
This also implements a fast-path that makes incremental diffs faster
when the log has not rolled over.
Related code cleanup.
If a slave is slow to consume what the master sends it, the master can
block and all iprop operations with it.
With minimal effort we make the master async oriented.
When the master sees a burst of updates (perhaps sustained), the
slaves "I_HAVE" messages can fall behind the version we've already
sent, and the unpatched code would retransmit already sent diffs!
This can result in substantial amplification (in a local test, 3000
ops turned into 427,000 ops). Though the number of *messages* sent
was actually somewhat smaller, the ever growing message size
ultimately leads to failure.
On a low update rate master, if we don't update old_version after
processing a poll timeout, we will generate spurious warnings about
missed (change) signals every time the timer expires, and will
needlessly contact the slaves.
When the master's log does not contain the complete history, slaves
that bootstrap from scratch encountered a loop, because the master
falsely assumed a race with log truncation.
- fix int/uint confusion and use unsigned integral types for time
- improve messages
- add --verbose option
- attempt transaction recovery in ipropd-master during idle times
- begin hardening daemons against dying at the slightest provocation
- better recovery from various errors
- daemons now restart automatically in most of the many error cases
where the daemons still die
We used to update the iprop log and HDB in different orders depending on
the kadm5 operation, which then led to various race conditions.
The iprop log now functions as a two-phase commit (with roll forward)
log for HDB changes. The log is auto-truncated, keeping the latest
entries that fit in a configurable maximum number of bytes (defaults to
50MB). See the log-max-size parameter description in krb5.conf(5).
The iprop log format and the protocol remain backwards-compatible with
earlier versions of Heimdal. This is NOT a flag-day; there is NO need
to update all the slaves at once with the master, though it is advisable
in general. Rolling upgrades and downgrades should work.
The sequence of updates is now (with HDB and log open and locked):
a) check that the HDB operation will succeed if attempted,
b) append to iprop log and fsync() it,
c) write to HDB (which should fsync()),
d) mark last log record committed (no fsync in this case).
Every kadm5 write operation recover transactions not yet confirmed as
committed, thus there can be at most one unconfirmed commit on a master
KDC.
Reads via kadm5_get_principal() also attempt to lock the log, and if
successful, recover unconfirmed transactions; readers must have write
access and must win any race to lock the iprop log.
The ipropd-master daemon also attempts to recover unconfirmed
transactions when idle.
The log now starts with a nop record whose payload records the offset of
the logical end of the log: the end of the last confirmed committed
transaction. This is kown as the "uber record". Its purpose is
two-fold: act as the confirmation of committed transactions, and provide
an O(1) method of finding the end of the log (i.e., without having to
traverse the entire log front to back).
Two-phase commit makes all kadm5 writes single-operation atomic
transactions (though some kadm5 operations, such as renames of
principals, and changes to principals' aliases, use multiple low-level
HDB write operations, but still all in one transaction). One can still
hold a lock on the HDB across many operations (e.g., by using the lock
command in a kadmin -l or calling kadm5_lock()) in order to push
multiple transactions in sequence, but this sequence will not be atomic
if the process or host crashes in the middle.
As before, HDB writes which do not go through the kadm5 API are excluded
from all of this, but there should be no such writes.
Lastly, the iprop-log(1) command is enhanced as follows:
- The dump, last-version, truncate, and replay sub-commands now have an
option to not lock the log. This is useful for inspecting a running
system's log file, especially on slave KDCs.
- The dump, last-version, truncate, and replay sub-commands now take an
optional iprop log file positional argument, so that they may be used
to inspect log files other than the running system's
configured/default log file.
Extensive code review and some re-writing for clarity by Viktor Dukhovni.
Tests that start daemons have to "wait" for them to start.
This commit makes Heimdal daemons prep to detach (when requested) by
forking early, then having the child signal readiness to the parent when
the child really is ready. The parent exits only which the child is
ready. This means that tests will no longer need to wait for daemons.
However, tests will still need a pidfile or such so they can stop the
daemons.
Note that the --detach options should not be used on OS X from launchd,
only from tests.
We should not hold locks on the master's database while waiting
for network I/O which may take a terribly long time to complete as
this will block out all writers and could therefore be slightly
problematic. ipropd-master was holding a shared lock on the database
while sending a complete propation to slaves which are out of sync
with the log file. We fix this by writing what we intend to send
in record format into a file hdb_db_dir()/ipropd.dumpfile while
holding a shared lock on the database and then we send the contents
of the file after releasing the lock. We also save and re-use the file
that we generated during future complete propagation events as long
as the log is long enough to get us back to the state previously
dumped.
We turn on a few extra warnings and fix the fallout that occurs
when building with --enable-developer. Note that we get different
warnings on different machines and so this will be a work in
progress. So far, we have built on NetBSD/amd64 5.99.64 (which
uses gcc 4.5.3) and Ubuntu 10.04.3 LTS (which uses gcc 4.4.3).
Notably, we fixed
1. a lot of missing structure initialisers,
2. unchecked return values for functions that glibc
marks as __attribute__((warn-unused-result)),
3. made minor modifications to slc and asn1_compile
which can generate code which generates warnings,
and
4. a few stragglers here and there.
We turned off the extended warnings for many programs in appl/ as
they are nearing the end of their useful lifetime, e.g. rsh, rcp,
popper, ftp and telnet.
Interestingly, glibc's strncmp() macro needed to be worked around
whereas the function calls did not.
We have not yet tried this on 32 bit platforms, so there will be
a few more warnings when we do.
The ipropd_slave will log its status to /var/heimdal/ipropd-slave-status
if its connecting, up to date, or disconnected.
The master will now also confirm to slaves that are are in fact up to date
if they just restart, before there was no confirmation, the slave just didn't
get any deltas.
Johan Gadsjö did a awesome analysis of the LDAP access pattens
and sent us a patch that reduced the calls the ldap server by 4
times as many. The patch was adopted and change to avoid compile
time depencies and make the determination runtime instead. Thanks!
