Actors never just leave the stage when their input is not being accepted, so why do musicians sometimes have to?
Your musicians are fellow players on stage with you. Be fully engaged with what they bring. Experience how they follow you, and know if and when you are following them.
Above all, listen to the music. Saying something like "Oh, the music was really great, I didn't even hear it!" is really not a compliment. Don't make your musician a second rate performer.
The audience notices the music. Make sure you do too.