Cabinet ministers' shrine visits spark backlash