Some propositions:
1. Freedom is a good thing. It is good in itself and it leads to good outcomes.
2. Freedom includes “free movement”.
3. Free movement is a bad thing. It leads to bad outcomes.
I can imagine some of the responses to this.
Freedom of movement is a success.
Really?

Anti-Jewish sit-in at Liverpool Street Station.
Still think it’s a success? I’d love to know what you would regard as failure.
Freedom of movement worked in the US in the 19th Century.
Yes, but not anywhere else. And certainly not here, and not now.
I know some great immigrants (and their descendants).
And so do I.

I’d go to war with her.

…and her.
The issue is not with those who come in small numbers. Or the ones who marry in. It’s with the ones who arrive en masse, live separately and learn to despise the natives.
There are problems but these would be solved with more freedom. If we abolished discrimination laws, hate speech laws etc things would be better. If we abolished planning (US=zoning) laws, the NHS and state education a lot of the pressures that immigration causes would be eased.
I am not sure that abolishing hate laws etc is even possible. People who find themselves mocked for their immutable characteristics are going to try to do something about it. Abolishing planning etc would be a good thing but that would do nothing to reduce the problems caused by mass migration. By making migration even more attractive it might even make them worse.
If you ditch freedom of movement where do you stop? freedom of speech, property rights?
That is the bit that troubles me the most. I want to believe that libertarianism has universal application. But what if it doesn’t? Here is an idea. Matters concerning the tribe are off-limits. Who is a member of the tribe? Where shall the tribe live? How shall the tribe defend itself? are simply outside the realm of libertarianism.
Update 5/11/23. When commenters started to mention the welfare state I had something of an “Oh drat!” moment. I’d simply forgotten to mention it. And it is a plausible explanation for both mass migration and its failure.
So, how do we assess the claim? We need to find examples of unsuccessful mass migration in the absence of a welfare state (or similar). This is not an easy thing to do. Welfare states and transport becoming affordable to even the world’s poorest came about at about the same time. There are a couple of counter-examples. Irish immigration to Belfast in the 19th Century for example. There were no Irishmen in Belfast before about 1800. There was no welfare state. There was lots of immigration to the shipyards and other industries. And by 1858 (if memory serves) there was lots of trouble. Another example which I can’t find was in a comment left here maybe 15 years ago. The commenter pointed out that Singapore had no welfare state, lots of immigration and ethnic tensions.