If non-primary host fails to process a request, retry on primary node.
conditions when we should not do this:
- no error
- we've aleady tried the primary
- we're making a status request to get the primary node...this
could lead to lock contention if we allow it to happen as we are
making an http request within an on going http request to discover
the primary node.
This also deletes the RemoveHost method and the associated test b/c
it is not used anywhere anymore and updates the returned error type.
