Use cases
=========

Startup:
    1. Blueprint everywhere is empty. No data is present anywhere.
    2. A blueprint is issued that designates one node as primary and two others
        as secondaries.
    3. The primary determines that since everything has version 0, it doesn't
        need to do a backfill. So it sets up a broadcaster_t, etc.
    4. The secondaries see that the primary is up and join it.

Add secondary:
    1. A primary and some secondaries are up.
    2. A blueprint is issued that designates an additional node as a secondary.
    3. The additional node sees that the primary is up and joins it.

Swap primary and secondary:
    1. A primary and some secondaries are up.
    2. A blueprint is issued that designates the old primary as a secondary and
        designates one of the old secondaries as a primary.
    3. The old primary sees that the new primary is live-tracking it, so it
        shuts itself off.
    4. The new primary sees that the old primary is lost, so it sets up a
        `broadcaster_t`, etc.
    5. The other secondaries see that the old primary is lost.
    6. The old primary and the other secondaries see that the primary is up and
        join it.

Move secondary:
    1. A primary and some secondaries are up.
    2. A blueprint is issued that removes an old secondary and adds a new one.
    3. The new secondary sees that the primary is up and joins it.
    4. The old secondary waits until the new secondary is up, then shuts itself
        off and deletes the data.

Swap primary and unused node:
    1. A primary and some secondaries are up.
    2. A blueprint is issued that designates a new node as the primary.
    3. The new primary sees that it's out-of-date and live-tracks the primary.
    4. When the new primary has caught up with the old primary, the old primary
        sees and shuts itself off.
    5. The new primary sees that the old primary is lost, so it sets up a
        `broadcaster_t`, etc.

Directory
=========

Purposes:
  * Advertising/finding broadcaster business cards
  * Advertising/finding backfiller business cards
  * Determining who to backfill from
  * Synchronization

Guarantees
==========

If A and B are each in the other's blueprint scope, they won't both have active
masters.

Procedure
=========

For all shards in the current blueprint:

    On condition:
        Blueprint designates us primary for that shard
        No overlapping masters are present in the directory
        We don’t have anything active for that shard
        Directory for this shard for every peer indicates Stable for some
            version less than or equal to our version
        We have received the directory-echo from the last blueprint from every
            peer in the blueprint's scope and the reply blueprint does not list
            any of them as primary for this shard or overlapping shards
    Do:
        Start up a broadcaster+master+listener+replier for that shard

    On condition:
        Blueprint designates us secondary for that shard
        We don't have anything active for that shard
        A master for that shard is present in the directory
    Do:
        Start up a listener+replier for that shard

    On condition:
        Blueprint designates us nothing for that shard
        We have some data for that shard
        We don't have anything active for that shard
        We have received the directory-echo from the last blueprint from every
            peer in the blueprint's scope and the reply blueprint lists them as
            secondary if our blueprint lists them as secondary

For all active broadcaster+master+listener+repliers:

    On condition:
        Blueprint does not designate us primary for that shard
        Something would react on this machine if we stopped
    Do:
        Stop the broadcaster+master+listener+replier

For all active listener+repliers:

    On condition:
        Blueprint does not designate us secondary or primary for that shard
        All secondaries mentioned in blueprint are in directory
        We have received the directory-echo from the last blueprint from every
            peer in the blueprint's scope and the reply blueprint lists them as
            secondary if our blueprint lists them as secondary
    Do:
        Stop the listener+replier

For all regions formed by taking a shard in the current blueprint and
    intersecting or subtracting regions from active components:

    On condition:
        Blueprint designates us primary for the shard
        We don’t have anything active for this region
        Directory for this region for some peer indicates Stable for some
            version greater than our version
    Do:
        Begin backfilling from that peer in that region

Procedure notes
===============

Backfills end on their own.

Listeners and repliers stop on their own if they lose contact with master.