-
Notifications
You must be signed in to change notification settings - Fork 129
feature: instance boot group and readiness #677
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
shwstppr
wants to merge
1
commit into
apache:main
Choose a base branch
from
shapeblue:feature-boot-group
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+354
−0
Draft
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
348 changes: 348 additions & 0 deletions
348
source/adminguide/virtual_machines/instance_boot_groups.rst
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,348 @@ | ||
| .. Licensed to the Apache Software Foundation (ASF) under one | ||
| or more contributor license agreements. See the NOTICE file | ||
| distributed with this work for additional information# | ||
| regarding copyright ownership. The ASF licenses this file | ||
| to you under the Apache License, Version 2.0 (the | ||
| "License"); you may not use this file except in compliance | ||
| with the License. You may obtain a copy of the License at | ||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
| Unless required by applicable law or agreed to in writing, | ||
| software distributed under the License is distributed on an | ||
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| KIND, either express or implied. See the License for the | ||
| specific language governing permissions and limitations | ||
| under the License. | ||
|
|
||
| About Instance Boot Groups | ||
| --------------------------- | ||
|
|
||
| An **Instance Boot Group** lets a User or Administrator group a set of Instances | ||
| and Instance Groups together so that they can be started, stopped, and | ||
| rebooted as a single, ordered unit. This is useful for multi-tier | ||
| applications where some Instances depend on others being up and reachable | ||
| before they should boot, for example a database tier that must be running | ||
| before an application tier starts, which in turn must be running before a | ||
| web tier starts. | ||
|
|
||
| An Instance Boot Group is owned by an account, just like an Instance, and is | ||
| managed from **Compute > Instance Boot Groups** in the CloudStack UI. | ||
|
|
||
| .. note:: An Instance Boot Group is a different concept from an Instance | ||
| Group (**Compute > Instance Groups**), which is a simple, unordered label | ||
| used to organize Instances. An Instance Group can itself be added as a | ||
| member of an Instance Boot Group. | ||
|
|
||
| Members and Boot Order | ||
| ~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| A **member** of an Instance Boot Group is either a single Instance or an | ||
| existing Instance Group. Each member is assigned a **Boot Order**, a | ||
| non-negative integer. | ||
|
|
||
| - All members that share the same Boot Order value form a **tier**, and boot | ||
| (or shut down) together, concurrently. | ||
| - Tiers are processed strictly in order: when starting a group, tier 0 is | ||
| processed before tier 1, tier 1 before tier 5, and so on. When stopping a | ||
| group, tiers are processed in the reverse order. | ||
| - An Instance or Instance Group can be a member of only one Instance Boot | ||
| Group at a time. | ||
| - An Instance that is part of a VNF appliance, or that already belongs to an | ||
| AutoScale VM group, cannot be added to an Instance Boot Group. | ||
| - The maximum number of members allowed in a single Instance Boot Group is | ||
| controlled by the domain-scoped configuration | ||
| ``instance.boot.group.max.members`` (default: 10). | ||
|
|
||
| Readiness Rules | ||
| ~~~~~~~~~~~~~~~~ | ||
|
|
||
| By default, CloudStack considers a member of an Instance Boot Group "ready" | ||
| as soon as its Instance (or, for an Instance Group member, all of its | ||
| Instances) reaches the Running state. For many applications this is not | ||
| enough — for example, a database Instance may be Running but not yet | ||
| accepting connections. | ||
|
|
||
| A **Readiness Rule** lets you define what "ready" really means for a member, | ||
| beyond the Instance's power state. When an Instance Boot Group is started, | ||
| CloudStack waits for a tier's readiness rules to report success before | ||
| moving on to start the next tier. | ||
|
|
||
| Readiness Rules can be attached directly to an Instance member or to an | ||
| Instance Group member. A rule attached to an Instance Group is automatically | ||
| **inherited** by every Instance currently in that group, and is evaluated | ||
| individually against each of them. | ||
|
|
||
| The following Readiness Rule types are supported: | ||
|
|
||
| .. cssclass:: table-striped table-bordered table-hover | ||
|
|
||
| ======================= ============================= ========================================================================================== | ||
| Rule Type Applies to Description | ||
| ======================= ============================= ========================================================================================== | ||
| Ping Instance, Instance Group Checks that the Instance responds to an ICMP ping, issued from the Virtual Router on the | ||
| Instance's default network. Requires a running Virtual Router on that network. | ||
| PortCheck Instance, Instance Group Checks that a TCP port on the Instance can be connected to, from the Virtual Router on the | ||
| Instance's default network. Requires the ``port`` detail (1-65535) and, optionally, the | ||
| ``protocol`` detail (only ``tcp`` is supported). | ||
| GuestAgentLiveness Instance, Instance Group Checks that the QEMU guest agent inside the Instance responds to a liveness ping. Supported | ||
| on **KVM only**. | ||
| MemberQuorum Instance Group only Considers the group ready once a threshold of its member Instances are ready, instead of | ||
| requiring all of them. Requires the ``thresholdtype`` detail (``COUNT`` or ``PERCENTAGE``) | ||
| and the ``thresholdvalue`` detail. | ||
| CustomScript Instance, Instance Group Reserved for a future release. It can be created, but currently always evaluates as an | ||
| error and is not offered in the CloudStack UI. | ||
| ======================= ============================= ========================================================================================== | ||
|
|
||
| .. note:: ``Ping``, ``GuestAgentLiveness``, and ``MemberQuorum`` are singleton | ||
| rules — only one of each can be attached to a given member. Multiple | ||
| ``PortCheck`` rules can be attached to the same member, one per port. | ||
|
|
||
| .. note:: ``Ping`` and ``PortCheck`` rules require the target Instance to | ||
| have a default network with a Virtual Router (they cannot be used on an | ||
| L2 network). | ||
|
|
||
| If a member has no Readiness Rules of its own and inherits none from an | ||
| owning Instance Group, its readiness is simply its Instance's power state | ||
| (Running or not). | ||
|
|
||
| Configuring Instance Boot Group Behaviour | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| The following global configuration parameters control how readiness is | ||
| checked while an Instance Boot Group is starting. All of them, except the | ||
| poll interval and the concurrency setting, can be overridden per Instance | ||
| Boot Group when it is created or updated. | ||
|
|
||
| .. cssclass:: table-striped table-bordered table-hover | ||
|
|
||
| ========================================================= ========================================================================================== | ||
| Configuration Description | ||
| ========================================================= ========================================================================================== | ||
| instance.boot.group.readiness.timeout.seconds Timeout, in seconds, for a single readiness check attempt. **Default: 300** | ||
| (overridable per Instance Boot Group) | ||
| instance.boot.group.readiness.max.retry.attempts Maximum number of readiness check attempts for a member before the tier, and the | ||
| Instance Boot Group start, is considered to have failed. **Default: 5** | ||
| (overridable per Instance Boot Group) | ||
| instance.boot.group.readiness.initial.delay.seconds Delay, in seconds, after an Instance is started before its first readiness check | ||
| attempt is made. **Default: 30** (overridable per Instance Boot Group) | ||
| instance.boot.group.readiness.reboot.on.retry Whether a member Instance should be rebooted before each new readiness check retry, | ||
| instead of simply being re-checked. **Default: false** (overridable per Instance Boot | ||
| Group) | ||
| instance.boot.group.readiness.poll.interval.seconds Interval, in seconds, between readiness check attempts. **Default: 10** (global only) | ||
| instance.boot.group.readiness.check.concurrency Maximum number of members within a tier whose readiness is checked concurrently. | ||
| **Default: 10** (global only) | ||
| instance.boot.group.max.members Maximum number of members allowed in a single Instance Boot Group. This is a | ||
| domain-scoped setting. **Default: 10** | ||
| ========================================================= ========================================================================================== | ||
|
|
||
| Creating an Instance Boot Group | ||
| --------------------------------- | ||
|
|
||
| #. Log in to the CloudStack UI. | ||
| #. In the left navigation bar, click **Compute**, then **Instance Boot Groups**. | ||
| #. Click **Add Instance Boot Group**. | ||
| #. Provide a name and, optionally, a description. | ||
| #. Optionally, expand the readiness settings to override the default | ||
| readiness timeout, maximum retry attempts, initial delay, or | ||
| reboot-on-retry behaviour for this Instance Boot Group only. | ||
| #. Click **OK**. | ||
|
|
||
| Using the API: | ||
|
|
||
| .. code:: bash | ||
|
|
||
| cmk create instancebootgroup name=web-app-stack description="DB, App and Web tiers" | ||
|
|
||
| Adding and Managing Members | ||
| ------------------------------ | ||
|
|
||
| Once an Instance Boot Group has been created, Instances or Instance Groups | ||
| can be added to it as members, each with its own Boot Order. | ||
|
|
||
| Adding a Member | ||
| ~~~~~~~~~~~~~~~~ | ||
|
|
||
| #. Open the Instance Boot Group and go to the **Members** tab. | ||
| #. Click **Add Member**. | ||
| #. Choose whether the member is an Instance or an Instance Group, then | ||
| select it from the list. | ||
| #. Enter the Boot Order for the new member. | ||
| #. Click **OK**. | ||
|
|
||
| Using the API: | ||
|
|
||
| .. code:: bash | ||
|
|
||
| # Add an Instance as a member of tier 0 (starts first) | ||
| cmk add membertoinstancebootgroup id=<boot-group-id> virtualmachineid=<vm-id> order=0 | ||
|
|
||
| # Add an Instance Group as a member of tier 1 (starts after tier 0) | ||
| cmk add membertoinstancebootgroup id=<boot-group-id> instancegroupid=<instance-group-id> order=1 | ||
|
|
||
| Changing a Member's Boot Order | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| From the **Members** tab, use the **Boot Order** action on a member row to | ||
| move it to a different tier. Any other members between the old and new | ||
| position are automatically shifted to keep the ordering consistent. | ||
|
|
||
| Using the API: | ||
|
|
||
| .. code:: bash | ||
|
|
||
| cmk update instancebootgroupmember id=<member-id> order=2 | ||
|
|
||
| Removing a Member | ||
| ~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| Removing a member from an Instance Boot Group only removes it from the | ||
| group (and deletes any Readiness Rules attached to it); the Instance or | ||
| Instance Group itself, and its running state, are not affected. | ||
|
|
||
| .. code:: bash | ||
|
|
||
| cmk remove instancebootgroupmember id=<member-id> | ||
|
|
||
| Listing Members and Their Status | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| The **Members** tab lists all members of an Instance Boot Group sorted by | ||
| Boot Order, along with a readiness status badge (Ready, NotReady, Error, or | ||
| Unknown) for each. An Instance Group member row can be expanded to see the | ||
| readiness of each Instance inside it. | ||
|
|
||
| .. code:: bash | ||
|
|
||
| cmk list instancebootgroupmembers bootgroupid=<boot-group-id> details=readiness,children | ||
|
|
||
| .. note:: Computing readiness details is not free, so it is only returned | ||
| when explicitly requested (``details=readiness``). Add | ||
| ``ignoreinstancestate=true`` to see the last cached rule result even | ||
| while the Instance is stopped; by default, a stopped Instance is always | ||
| reported NotReady regardless of the cached result. | ||
|
|
||
| Configuring Readiness Rules for a Member | ||
| ------------------------------------------- | ||
|
|
||
| #. From the **Members** tab, use the **Manage Readiness Rules** action on | ||
| the member you want to configure. | ||
| #. Click **Add Rule**, choose a Rule Type, and fill in the fields relevant | ||
| to that type (for example, the port and protocol for a ``PortCheck`` | ||
| rule, or the threshold type and value for a ``MemberQuorum`` rule on an | ||
| Instance Group). | ||
| #. Click **OK**. | ||
|
|
||
| Rules inherited from an owning Instance Group are shown read-only, tagged | ||
| **Inherited**; they must be removed from the Instance Group member instead. | ||
|
|
||
| .. note:: When adding a ``GuestAgentLiveness`` rule, make sure the QEMU | ||
| guest agent is installed, running, and responsive inside the guest | ||
| Instance — the rule can only report Ready once the agent answers. | ||
|
|
||
| Using the API: | ||
|
|
||
| .. code:: bash | ||
|
|
||
| # A TCP port check on an Instance | ||
| cmk create instancebootgroupreadinessrule bootgroupid=<boot-group-id> \ | ||
| virtualmachineid=<vm-id> ruletype=PortCheck \ | ||
| details[0].port=1433 details[0].protocol=tcp | ||
|
|
||
| # A member quorum rule on an Instance Group (ready once 75% of its Instances are ready) | ||
| cmk create instancebootgroupreadinessrule bootgroupid=<boot-group-id> \ | ||
| instancegroupid=<instance-group-id> ruletype=MemberQuorum \ | ||
| details[0].thresholdtype=PERCENTAGE details[0].thresholdvalue=75 | ||
|
|
||
| cmk list instancebootgroupreadinessrules bootgroupid=<boot-group-id> | ||
|
|
||
| cmk update instancebootgroupreadinessrule id=<rule-id> enabled=false | ||
|
|
||
| cmk delete instancebootgroupreadinessrule id=<rule-id> | ||
|
|
||
| .. note:: The rule type of an existing Readiness Rule cannot be changed; | ||
| only its name, enabled state, and details can be updated. Delete and | ||
| re-create the rule to change its type. | ||
|
|
||
| Starting, Stopping, and Rebooting an Instance Boot Group | ||
| ------------------------------------------------------------- | ||
|
|
||
| Instance Boot Groups are started, stopped, and rebooted as a whole from the | ||
| **Instance Boot Groups** list or detail view, or via the API. These are | ||
| asynchronous operations. | ||
|
|
||
| - **Start**: tiers are started in ascending Boot Order. All members of a | ||
| tier are started concurrently, and CloudStack waits for the tier to | ||
| become ready — based on its members' Readiness Rules — before starting | ||
| the next tier. If a member does not become ready within the configured | ||
| number of retry attempts, or fails to start, the whole start operation is | ||
| halted. Instances that were already started before the failure are left | ||
| running; they are not automatically stopped again. | ||
| - **Stop**: tiers are stopped in descending Boot Order, the reverse of | ||
| start. Readiness Rules are not checked while stopping, and a failure to | ||
| stop one member does not prevent CloudStack from continuing to stop the | ||
| remaining tiers. | ||
| - **Reboot**: equivalent to a stop followed by a start, i.e. the whole | ||
| group is stopped in descending order and then started again in ascending, | ||
| readiness-gated order. | ||
|
|
||
| Using the API: | ||
|
|
||
| .. code:: bash | ||
|
|
||
| cmk start instancebootgroup id=<boot-group-id> | ||
|
|
||
| cmk stop instancebootgroup id=<boot-group-id> forced=true | ||
|
|
||
| cmk reboot instancebootgroup id=<boot-group-id> | ||
|
|
||
| Deleting an Instance Boot Group | ||
| ------------------------------------ | ||
|
|
||
| Deleting an Instance Boot Group removes its members and any Readiness | ||
| Rules attached to them; the Instances and Instance Groups themselves are | ||
| left untouched and continue running or stopped as they were. | ||
|
|
||
| .. code:: bash | ||
|
|
||
| cmk delete instancebootgroup id=<boot-group-id> | ||
|
|
||
| Instance Boot Group Events | ||
| --------------------------------- | ||
|
|
||
| The following events are logged for auditing and monitoring Instance Boot | ||
| Group activity: | ||
|
|
||
| .. cssclass:: table-striped table-bordered table-hover | ||
|
|
||
| ================================================= ====================================================== | ||
| Event Type Description | ||
| ================================================= ====================================================== | ||
| INSTANCE.BOOT.GROUP.CREATE An Instance Boot Group was created | ||
| INSTANCE.BOOT.GROUP.UPDATE An Instance Boot Group was updated | ||
| INSTANCE.BOOT.GROUP.DELETE An Instance Boot Group was deleted | ||
| INSTANCE.BOOT.GROUP.START An Instance Boot Group was started | ||
| INSTANCE.BOOT.GROUP.STOP An Instance Boot Group was stopped | ||
| INSTANCE.BOOT.GROUP.REBOOT An Instance Boot Group was rebooted | ||
| INSTANCE.BOOT.GROUP.MEMBER.ADD A member was added to an Instance Boot Group | ||
| INSTANCE.BOOT.GROUP.MEMBER.REMOVE A member was removed from an Instance Boot Group | ||
| INSTANCE.BOOT.GROUP.READINESS.RULE.CREATE A Readiness Rule was created | ||
| INSTANCE.BOOT.GROUP.READINESS.RULE.UPDATE A Readiness Rule was updated | ||
| INSTANCE.BOOT.GROUP.READINESS.RULE.DELETE A Readiness Rule was deleted | ||
| ================================================= ====================================================== | ||
|
|
||
| Limitations | ||
| ------------------ | ||
|
|
||
| #. An Instance or Instance Group can belong to at most one Instance Boot | ||
| Group at a time. | ||
| #. Instances that are part of a VNF appliance, or already belong to an | ||
| AutoScale VM group, cannot be added to an Instance Boot Group. | ||
| #. The ``GuestAgentLiveness`` Readiness Rule requires the QEMU guest agent | ||
| and is supported on KVM Instances only. | ||
| #. ``Ping`` and ``PortCheck`` Readiness Rules require a running Virtual | ||
| Router on the Instance's default network, and cannot be used on Instances | ||
| connected only to an L2 network. | ||
| #. The ``CustomScript`` Readiness Rule type is reserved for a future | ||
| release; it is not yet implemented and is not available from the UI. | ||
| #. If starting an Instance Boot Group is halted because a tier fails to | ||
| become ready, any Instances already started in earlier tiers are left | ||
| running — CloudStack does not automatically roll the operation back. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's repeating, either keep heading here or in
source/adminguide/virtual_machines.rst