You are not responsible for the scene; the group is. You are responsible your partners on stage.
Pretty self explanatory, isn't it? The concept of group responsibility is really core to improv. No individual can or may make all of the big decisions.
Mind that you're not even responsible for yourself. The others are! You are only responsible to take care of your partners.