Ilminster Carnival organisers say fundraising has struggled due to the visit of the Prince of Wales.