If you have even only one good improviser on stage, nobody on that stage is a bad improviser.
We have a huge advantage over scripted theatre, because everything we do happens for the first time, for them, with their suggestions. Our audiences are much more invested in our work.
Also 'yes, and' randomness. Expand your circle of expectation to contain weirdness.
You're not responsible for everything you imagination comes up with. It doesn't define your personality.