Can't use the broom while the effect is active.
You may need to accept the quest "Hallowtide Broom Race" again.