Religious freedom is one of those basic rights that I've always had a hard time getting my head around.
On the one hand, I get the idea that it's an extension of freedom of association. Of course people should be allowed to form whatever kind of groups they choose, within some very broad limits, if any limits at all. Religions are groups, even if they're rather crazy, so governments should leave them alone.
This doesn't mean, of course, that the rest of us have to leave them alone. Just that we need to be careful not to try to get the legal of political institutions involved. It's one thing to say that Christianity is a cult and people shouldn't be involved in it; it's quite another to say that people should be jailed or sued for being Christians.
On the other hand, there's reason to be suspicious of the idea that religion is really analogous to voluntary associations of other kinds. Religion is more like culture, in that one is often raised in a certain religious tradition, and thus one is inclined to take its claims at face value. This leads to various forms of degradation, both moral and cognitive.
For example, there are religions which treat women as being inferior (because of the will of the divine!) to men. Women raised in these religions who take them seriously will run into a standard cheap tastes situation: having their preferences limited and restricted to serving men. Similarly, men raised in these religions who take them seriously will end up with pretty distorted characters: treating women's ideas and contributions as inherently less valuable than men's, regardless of their substance. (Not to say these two problems are equivalent, only that they are of the same kind.)
On the cognitive end, I think it's fairly clear that the deeply religious have problems with basic epistemic virtues, particularly when it comes to considering the truth of claims which conflict with the tenets of the religion. The rational thing to do is to weigh the two against each other, and give up whichever violates the most epistemic values (such as consistency, evidential support, fecundity of prediction, and so on). Having any beliefs which one would never give up, under any circumstances, is a pretty serious cognitive deficit.
There's also the problem of children. It would be one thing if adults willingly submitted themselves to moral and cognitive degradation. There's an argument to be made here basd on Mill's Harm Principle, that no legal or political intervention is justified except insofar as it prevents harm to others. Adults who are only harming themselves through their conduct can be remonstrated with, bribed, pleaded with, and so on, but never legally coerced.
When it comes to children, though, who cannot be held fully responsible for their behaviour, things get more complicated. It may not be child abuse to raise a child in a moderate form of Christianity or Islam, but what about raising a child as a Scientologist? Or Amish?
This is an issue I've never been able to resolve to my own satisfaction. I can see both arguments, and it looks like a clear conflict between my libertarian tendencies -- religion is just another group -- and my socialist tendencies -- religion is damaging to many people.