It depends. There are certain stipulations. The question is too general and geared towards the mid-middle class and above. Poorer populations would, of course, say "price", and "brand name" becomes an essence of desire rather than justification.
Since your question is appropriate for the mid-middle class and above population, and I fit in this category (humbly speaking), I choose to go for price.
And to tell you the truth, quality has little significance to the rationalization/reason. Do you know why? Well, the fact is quality exist in both brand/non-brand (or cheap) names. Both contains good quality and BAD quality. I refrain from using the term "cheap". I like "affordable" because then I don't rule out the...poor population because they are people too. "Affordable" clothes are not ALL poor in quality. Expensive/brand name clothes are not ALL great quality (I can say the same about the character of people). In fact, I have a few brand names, and, actually, more NON-brand name clothings. BUT, they're all good quality. I have some clothes that are still in pretty damn good shape after 2 years of the purchase date.
THEREfore, brand name or not, it's all about careful shopping. Cotton is cotton. Polyester is polyester.