After the Council or Board shall have established building or setback lines on such major highways, no new building or structure shall be erected within such building or setback lines in the area and no permit for such building shall be issued by the City Building Inspector as respects the territory over which the municipality shall have enforcement jurisdiction or by the County Inspecting Officer appointed by the Board to administer and enforce building line and setback regulations and other regulations for the county, as respects the territory over which the county shall have enforcement jurisdiction. The City Board of Adjustment, as respects the territory over which the municipality shall have enforcement jurisdiction, and the County Board of Adjustment as respects the territory over which the county shall have enforcement jurisdiction, shall hear appeals of any property owner aggrieved by such building line and setback regulations. Each said Board of Adjustment shall have the power to modify or vary the building line or setback regulations in specific cases, in order that unwarranted hardship, which constitutes an unreasonable deprivation of uses as distinguished from a mere grant of privilege, may be avoided, the intended purpose of the regulations strictly observed, and the public welfare and public safety protected.