How do we know that the color blind people in this world aren't right about what they see, and we're not?

Answer 1

People should be able to see the colors in the visible spectrum. The visible spectrum can be measured by instruments other than sight based ones.
Color blindness is a problem in visual perception. So it doesn't matter really who's right or wrong, does it?

Answer 2

Color blind people can't distinguish between certain pairs of color that other people can. It is a handicap, not just a difference. Imagine a rainbow. To most people it has a range of completely different colors. To a color blind person, it might appear that a couple of the bands, although not next to each other, are the same color. So who is right?
Of course none of us can see ultraviolet or infrared, but many insects can, so they might think all humans are partially color blind.

Answer 3

A common form of color blindness is red/green in which the person is unable to reliably tell the difference between red and green light.

So get a red light and green light, or at least a lights that a "normally" sighted person says are red and green. Determine that the color blind person cannot tell the difference. Shine a beam of each into a prism and measure the angle of refraction. If you get two different angles, you have two different wavelengths and hence two different colors. If one person can tell the difference and another cannot, guess who is right?

Answer 4

How do we even know that one color looks the same from each person to the next?

Color blindness, in it's most common form, results from an inability of the red and green cones in your eyes to distinguish between the two. Red and green are detected by the same cones. It's hard to explain in a short answer. You should read up on it.