Priests and Collars

I am a Catholic and a couple years ago, my neighbors (who are also Catholic) invited me over for a party one Sunday night. There were a lot of people present. Among the people present, my neighbors' parish priest was present. Although I really didn't care too much, I happen to notice that he was wearing layman/street clothing (polo shirt and khaki pants if I remember correctly) instead of a clerical collar. I told my mother about the party and brought up the fact that the priest had come over for the party. She then asked me, "How did you know that he was a priest, was he wearing a collar?" And then I said "No he wasn't." She then told me that the priest was committing a grave sin because the Church "supposedly" teaches that all clerics (deacons, priests, bishops) are forbidden to wear street clothing and are required to either wear a cassock or a collar at all times to show that they are ordained clerics in the Church.

I have never really understood this concept, is my mother right or is she worng? And what is the official position of the Church in terms of clerical dress?