It's impossible to shake off this temptation.
Your Fishing proficiency is increased while you are charmed.