A former town councillor feels ignored amid the construction of a special needs school next to his caravan site.