You made the right decision to go home. You can shoot when things warm up.
I stayed up in my detached garage as the temperature went to "minus 8". I wasn't shooting. I was just enjoying the quiet solitude. I had endured these kinds of temperatures when I was a teenager and I thought I was still good. I was well-layered and didn't feel cold. But by the time I made it back to the house, I could barely walk. I fell down in the living room and my wife struggled to get me back up. I am in the "doghouse" now for scaring her. I made the wrong decision, you made the right one.