If only the judges can vote him off the show