No shoes, no shirt isn't the same as "no fags".
Primarily public health and safety, and in most places it's just a matter of common decency. There are settings where no shirt would be perfectly acceptable. Marinas, beachside establishments, and other settings where people frequent without shirts.