It's too hard socially
Social friction is genuine, particularly at first. It is a cost to weigh rather than a reason the underlying case fails.
Food is woven into nearly every part of social life: family holidays, dinners with friends, work events, dating, travel, and the dishes that carry a family's or a culture's identity. Going vegan means turning yourself into a standing complication at all of these, explaining and defending your choice again and again, straining relationships with people you love, and living under scrutiny for every slip. A demand that costs this much in belonging and goodwill is too much to ask of ordinary people, and it is fair to weigh that cost heavily rather than treat it as a minor inconvenience.
The difficulty is real and worth naming honestly: explaining yourself repeatedly, being the complication at someone else's dinner, family tension, and the sense of being watched for a slip. For some people this is the hardest part, considerably harder than the food.
It bears on how difficult the course of action is, not on whether it is right, and the two keep getting conflated. Practically, it also eases. Restaurant options have improved sharply, most people are indifferent rather than hostile once the novelty passes, and the friction is loudest in the first few months.
Introducing a point that is not relevant to the question under discussion, drawing attention away from it rather than answering it.
How it applies here: Difficulty bears on how demanding a course of action is, not on whether it is justified. The objection answers a question of cost with the vocabulary of a question about rightness.
Preferring existing arrangements simply because they are the existing arrangements, and treating any change as carrying a burden of proof that the present state does not.
How it applies here: The awkwardness of departing from a norm is treated as a property of the alternative rather than of the norm. The same friction would attach to any majority practice.