in which case we prefer unstable control system instead of stable control system?

Answer 1

Sounds like an oxymoron. I'm not that familiar with the terminology of control systems, but one possibliy is that an "unstable" system may react faster, and if speed is a primary requirement, then it might be prefered. Another thought is something like a rupture disk for pressure relief, I don't know that would be called a "control system", but once things are so bad you need to blow the disk, you may not care so much if you vent the whole system.

Answer 2

you never need an unstable control system
because by definitionit will never return to an equilibrium state
renedering it useless
only reason for unstabilising a system is to destroy it
Now, ther are systems which are designed to be nearly unstable or statically unstable and are being stabilised all the time by Adaptive control
Such systems give very high performance (like high manoverability in)
Some birds or maybe most are unstable but they stabilise themselves all the time.

Answer 3

you will only seriously consider an unstable control if the results for the stable just can't work and you have a nearby human operator to always keep an eye on the thing....

Hence, unstable over your boat isn't that bad, if someone always brings it back in check before deviations get ugly....

Answer 4

You don't prefer an unstable one. It will oscillate and go from it's max to its min. Imagine the cruise control in your car working that way.