If you're missing some specific game, others that do have it will still be able to play it, and only you will see a blank TV for that game. You don't need to have all the same games.
You can pass the controller on the fly and anyone can continue the game where you left while you watch them.