Minister to quit over Olympic woes