I suppose they would have to be explained using the other senses as a basis.
Brown would be a very nature-ish color, the color of wood and dirt and fallen leaves. The color of most animals and of earth.
Blue would be water, cool and soothing.
I think purple is the only color I can't really think of much to describe, unless it's the rich, luxurious feel of velvet.