Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ iproto:

replication:
failover: supervised
bootstrap_strategy: native

failover:
probe_interval: 5
Expand Down
17 changes: 11 additions & 6 deletions doc/platform/replication/supervised_failover.rst
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,16 @@ The main steps of using an external failover coordinator for a newly configured

1. :ref:`Configure a cluster <supervised_failover_configuration>` to work with an external coordinator.
The main step is setting the ``replication.failover`` option to ``supervised`` for all replica sets that should be managed by the external coordinator.
For newly configured clusters, set :ref:`replication.bootstrap_strategy <configuration_reference_replication_bootstrap_strategy>` to ``native`` so that the coordinator controls the initial bootstrap as well.

2. Start a configured cluster.
When an external coordinator is still not running, instances in a replica set start in the following modes:
If :ref:`replication.bootstrap_strategy <configuration_reference_replication_bootstrap_strategy>` is set to ``native`` or ``supervised``, an unbootstrapped replica set waits for the failover coordinator to appoint a bootstrap leader and no instance bootstraps on its own.
After bootstrap, instances are started in read-only mode until the coordinator appoints a leader.

- If a replica set is already :ref:`bootstrapped <replication_stages>`, all instances are started in read-only mode.
- If a replica set is not bootstrapped, one instance is started in read-write mode.
If the default ``auto`` bootstrap strategy is used instead, one instance in an unbootstrapped replica set can start in read-write mode before the coordinator starts.

3. :ref:`Start a failover coordinator <supervised_failover_start_coordinator>`.
If the replica set is waiting for a bootstrap leader, start the coordinator before :ref:`replication.connect_timeout <configuration_reference_replication_connect_timeout>` expires, or start the coordinator before the instances.
You can start two or more failover coordinators to increase fault tolerance.
In this case, one coordinator is active and others are passive.

Expand Down Expand Up @@ -108,12 +110,12 @@ To configure a cluster to work with an external failover coordinator, follow the

#. (Optional) If you need to run :ref:`several failover coordinators <supervised_failover_overview_fault_tolerance>` to increase fault tolerance, set up an etcd-based configuration storage, as described in :ref:`configuration_etcd`.

#. Set the :ref:`replication.failover <configuration_reference_replication_failover>` option to ``supervised``:
#. Set the :ref:`replication.failover <configuration_reference_replication_failover>` option to ``supervised`` and :ref:`replication.bootstrap_strategy <configuration_reference_replication_bootstrap_strategy>` to ``native``:

.. literalinclude:: /code_snippets/snippets/replication/instances.enabled/supervised_failover/source.yaml
:language: yaml
:start-at: replication:
:end-at: failover: supervised
:end-at: bootstrap_strategy: native
:dedent:

#. Grant a user used for replication :ref:`permissions <configuration_credentials_managing_users_roles_granting_privileges>` to execute the ``failover.execute`` function:
Expand All @@ -126,14 +128,17 @@ To configure a cluster to work with an external failover coordinator, follow the

.. note::

When ``replication.bootstrap_strategy`` is ``native`` or ``supervised``, Tarantool automatically grants the ``guest`` user runtime privileges to execute ``failover.execute`` for the initial cluster bootstrap.
The explicit grant above lets the coordinator use the configured replication user after the cluster is bootstrapped.

In Tarantool 3.0 and 3.1, the configuration is different and the function
must be created in the application code. See :ref:`supervised_failover_configuration_with_role` for details.

#. (Optional) Configure options that control how a failover coordinator operates in the :ref:`failover <configuration_reference_failover>` section:

.. literalinclude:: /code_snippets/snippets/replication/instances.enabled/supervised_failover/source.yaml
:language: yaml
:start-after: failover: supervised
:start-after: bootstrap_strategy: native
:end-before: groups
:dedent:

Expand Down
5 changes: 5 additions & 0 deletions doc/reference/configuration/cfg_replication.rst
Original file line number Diff line number Diff line change
Expand Up @@ -264,6 +264,11 @@
* ``legacy`` (deprecated since :doc:`2.11.0 </release/2.11.0>`): a node requires the :ref:`replication_connect_quorum <cfg_replication-replication_connect_quorum>` number of other nodes to be connected.
This option is added to keep the compatibility with the current versions of Cartridge and might be removed in the future.

.. NOTE::

The ``native`` strategy is available only in the YAML configuration as :ref:`replication.bootstrap_strategy <configuration_reference_replication_bootstrap_strategy>`.
When ``native`` is configured in YAML, Tarantool uses ``supervised`` as the underlying ``box.cfg`` bootstrap strategy and manages the bootstrap leader in the configuration layer.

|
| Type: string
| Default: auto
Expand Down
20 changes: 20 additions & 0 deletions doc/reference/configuration/configuration_reference.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3995,10 +3995,30 @@ autoexpel full example
To specify the bootstrap leader, use the :ref:`<replicaset_name>.bootstrap_leader <configuration_reference_replicasets_name_bootstrap_leader>` option.

* ``supervised``: a bootstrap leader isn't chosen automatically but should be appointed using :ref:`box.ctl.make_bootstrap_leader() <box_ctl-make_bootstrap_leader>` on the desired node.
The bootstrap leader management is the user's responsibility unless :ref:`replication.failover <configuration_reference_replication_failover>` is set to ``supervised``.
In this case, the failover coordinator manages the bootstrap leader.

* ``native`` (since :doc:`3.4.0 </release/3.4.0>`): the bootstrap leader is managed in the YAML configuration layer in sync with read-only/read-write mode management.
From the user's perspective, this strategy is similar to ``auto``: Tarantool or the failover coordinator handles bootstrapping automatically.
Internally, ``native`` is based on ``supervised``, which helps avoid limitations of ``auto``, such as requiring a joining replica to connect to all instances registered in the ``_cluster`` system space.

When initializing a replica set, ``native`` uses :ref:`box.ctl.make_bootstrap_leader({graceful = true}) <box_ctl-make_bootstrap_leader>`.
After bootstrap, when an instance switches to read-write mode, ``native`` uses :ref:`box.ctl.make_bootstrap_leader() <box_ctl-make_bootstrap_leader>` to keep the bootstrap leader record pointing to the current read-write instance.
The instance chosen to initialize a replica set depends on :ref:`replication.failover <configuration_reference_replication_failover>`:

- ``off``: the first read-write instance in lexicographic order.
- ``manual``: the instance configured as the replica set leader.
- ``election``: the first non-anonymous candidate in lexicographic order.
- ``supervised``: the failover coordinator chooses the bootstrap leader.

* ``legacy`` (deprecated since :doc:`2.11.0 </release/2.11.0>`): a node requires the :ref:`replication_connect_quorum <cfg_replication-replication_connect_quorum>` number of other nodes to be connected.
This option is added to keep the compatibility with the current versions of Cartridge and might be removed in the future.

.. NOTE::

When using ``supervised`` or ``native`` with :ref:`replication.failover <configuration_reference_replication_failover>` set to ``supervised``,
Tarantool automatically grants the ``guest`` user runtime privileges to execute the internal ``failover.execute`` call for the initial cluster bootstrap.

|
| Type: string
| Default: ``auto``
Expand Down
21 changes: 18 additions & 3 deletions doc/reference/reference_lua/box_ctl/make_bootstrap_leader.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,33 @@
box.ctl.make_bootstrap_leader()
===============================

.. function:: make_bootstrap_leader()
.. function:: make_bootstrap_leader([opts])

**Since:** :doc:`3.0.0 </release/3.0.0>`.

Make the instance a bootstrap leader of a replica set.

To be able to make the instance a bootstrap leader manually, the :ref:`replication.bootstrap_strategy <configuration_reference_replication_bootstrap_strategy>` configuration option should be set to ``supervised``.
To be able to make the instance a bootstrap leader manually, the effective bootstrap strategy should be ``supervised``.
This is the case when ``box.cfg{bootstrap_strategy = 'supervised'}`` is used or when :ref:`replication.bootstrap_strategy <configuration_reference_replication_bootstrap_strategy>` in the YAML configuration is set to ``supervised`` or ``native``.
With ``native``, the configuration logic normally calls ``box.ctl.make_bootstrap_leader()`` automatically.

In this case, the instances do not choose a bootstrap leader automatically but wait for it to be appointed manually.
Configuration fails if no bootstrap leader is appointed during a :ref:`replication.connect_timeout <configuration_reference_replication_connect_timeout>`.

:param table opts: optional parameters. The following option is available:

- ``graceful`` (boolean) -- if ``true``, request a graceful bootstrap.
If the local database is absent, the instance checks whether a bootstrap leader exists on connected peers and takes this role only if no bootstrap leader is found.
If the local database already exists, the call is a no-op.
The default value is ``false``.

.. NOTE::

The ``graceful`` option is available since :doc:`3.4.0 </release/3.4.0>`.
It doesn't protect an unbootstrapped replica set from two instances that simultaneously try to bootstrap it gracefully.

.. NOTE::

When a new instance joins a replica set configured with the ``supervised`` bootstrap strategy,
When a new instance joins a replica set configured with the ``supervised`` or ``native`` bootstrap strategy,
this instance doesn't choose the bootstrap leader automatically but joins to the instance on which
``box.ctl.make_bootstrap_leader()`` was executed last time.