Holster promblems

I've been carrying my 5.7 in my crossbreed iwh lately. I've noticed that occasionally my safety is on fire when I unholster it. So today I make sure it was unloaded and carried it all day testing it. In and out of truck , casual walking and normal stuff. 6 times when I pulled it out it was on fire. I cannot get it to do on purpose. I wiggle, shake, bend gun in all directions and nothing. Stays on safety. The first two times I noticed it the gun was loaded and chambered a few weeks ago. Any ideas or has this happened to anyone else?