in adobe photoshop what check box creates a smooth edge that can apply to both selecting and erasing.?

Answer 1

There is no "check box" to smooth edges.
It needs to be done manually, or by using some of the tools such as the blur tool.

You might be thinking of Illustrator to give you a nice smooth line, but then again, they are 2 different programs for 2 different kinds of work.

Photoshop is for pixel based images (photos).
Illustrator is for vecot based images (lines).