Fix downstream endpoint recovery

[faizana-nvidia]

This change introduces mechanism to trigger recovery for endpoints which are behind the bridge. Current Recovery dbu-method doesn't work if triggered for downstream endpoint due to lack to right query path (addressed to downstream endpoint) with this PR we aim to merge with existing logic while handling downstream endpoint use case as well. Originally issue was being tracked via #149

[jk-ozlabs]

Thanks for the PR. One main thing to start:

1. Physical liveness check: send Get Endpoint ID to the bridge's
   hardware address.
2. EID-routed probe: send Get Endpoint ID via EID routing through the
   bridge to verify the downstream device is still present.
3. UUID check via EID routing: compare the UUID returned by the
   downstream device against the stored UUID to detect device exchange.
   A UUID mismatch triggers InterfacesRemoved + InterfacesAdded so the
   new device is registered cleanly.

This seems a bit backwards - I would have expected that we perform the Get Endpoint ID query to the endpoint first, to establish base connectivity. If that fails, then walk up the tree to find the top-most bridge with the connectivity issue.

Is there a reason to interact with the bridge first?

[faizana-nvidia]

Is there a reason to interact with the bridge first?

The way I was looking into this was, since everything will be routed via bridge if bridge itself is non responsive then probably downstream path might be broken too and we could be seeing timeouts with probe on downstream EID.

[jk-ozlabs]

The way I was looking into this was, since everything will be routed via bridge if bridge itself is non responsive then probably downstream path might be broken too and we could be seeing timeouts with probe on downstream EID.

Yes, I understand the rationale for checking connectivity to the bridge, but I think the ordering should be reversed - that we want to check for the smallest failure first - being the actual endpoint. If we have lost connectivity there, then we can walk up the bridge topology. Keep in mind there may be multiple bridges to walk!

Also: a call to Recover may not mean we have actually lost connectivity to the downstream endpoint at all, just that something has gone wrong at the application level, and that mctpd should query it.

[faizana-nvidia]

Ack, I'll update the logic to change the order. Also when you mention about multiple bridge walk, I do get the topological aspect and wanted to find out which bridge comms exactly broke, but cascaded bridge system support is not yet added, so may be we can delegate it for later?

[jk-ozlabs]

but cascaded bridge system support is not yet added, so may be we can delegate it for later?

Yes, but you don't need to delegate much - at present, it will just appear as though the top-level bridge of a cascade just hosts all downstream devices (ie,. we do not have visibility on the intermediate bridges with the current code).

So, once we have a concept of those intermediate bridges, as long as the recovery code has a "query the peer's bridge" path, we would just change that to iterate over the tree (ie, to the peer's bridge's bridge, etc) as suitable.

However: with the intermediate bridges missing from the topology, we may see the following case:

graph TD;
    TMBO-->B1[Bridge 1];
    B1--> B2[Bridge 2];
    B2--> EP;

Loading

If bridge 2 is offline, the EP will be unreachable, but the only bridge that the TMBO is aware of is Bridge 1, which will appear to have full connectivity. We'll need to handle that case gracefully.

[faizana-nvidia]

Ack, I see two ways of doing this.

1. Assuming our peer->pool_start and peer->pool_end is maintained in our code down to every bridge in the path.
   Bridge1 [PS1, PE1], Bridge2[PS2, PE2],..

Although I'm not sure how could this be maintained since there is probably no direct way of asking bridge about its allocation results (Pool Start, Pool End) apart from what TBO knows about its nearest Bridge, but if this is somehow maintained then :

We can recursively trace each bridges which might contain the endpoint eid E i.e (PSx < E < PEx) from TBOs nearest and check bridge's connectivity till we reach the last bridge nearest to Endpoint E. If intermediate bridge is point of failure then simply flag that Bridge EID as break point, may be?

2. Introduce Query Hop may be to systematically ask for nearest Bridge for given endpoint EID and then check connectivity to that bridge eid, all the way down till query hop successfully returns next bridge eids