Why can’t DNA polymerase go 3’ -> 5’?

Answer 1

It's putting round pegs in square holes; the DNA wouldn't fit into the DNA polymerase if it went in that direction. It's the same reason why you can't put the left shoe on the right foot.
Proteins function off of shapes, and if the molecule the protein is trying to act on doesn't fit that specific protein, then the protein won't be able to do anything.

Answer 2

First answer is right; polymerases have a definate structure and they only recognize DNA from 5' to 3'. No 3' to 5' polymerases have been found in nature, incredibly enough. I'm not sure why that hasn't evolved, but it hasn't. Reverse transcriptase can go 3' to 5' though.