There is no penalty for losing a minigame: you can either try again or simply skip it.