Luxembourg is smaller than Rhode Island, the smallest state in the US. It is about 1/70 the size of Oklahoma and less than .01% of the size of the US as a whole. It's about twice the size of the EU as a whole. In terms of population, the population of Luxembourg is less than that of Alaska, and less than 1/3 of one percent of the US as a whole.
Even if mandatory voting would not be unconstitutional (it most likely would, as not voting would be interpreted as a form of speech protected by the first amendment), and would lead to a more educated electorate (I'm skeptical of this), it would simply be a logistical nightmare. Things that may work out in a small country with a centralized population might not work in a country like the US.
Keep in mind that mandatory voting also holds true for European elections, so your size comparisons are not exactly true here.
As for your second point, if Brazil can do it and enforce it through the deepest jungle, why shouldn't the US be able to do so if it's constitutional?
According to the EU's official website...
"Voting is compulsory in Belgium, Bulgaria, Cyprus, Greece, and Luxembourg."
Greece is the biggest of those, and is about the size of Louisiana. I believe all of them could fit within the state of Montana.
As for the second point, I don't know a whole lot about Brazil and their elections, so it's hard to compare directly. I read that the voting rates in Brazil are about 80%, which is way higher than the US, but shows that mandatory voting doesn't get you close to 100%. And, with the minor fine assessed, it seems like the cost would probably outweigh the benefit.
The biggest issue though would be the US' federalist system. Protecting voting is typically a state power, and any federal laws would be unconstitutional. There are ways to get around that (for instance by using federal funds as a carrot to get states to agree to voting reform), but there would be a lot of legal challenges in the US to establishing mandatory voting on a federal level. I don't know if that would apply in Brazil.
States might be able to mandate voting, but that also might be unconstitutional. In Brazil, you can send back a blank ballot which may alleviate the problem, but the First Amendment is read very broadly, and even limiting the option of not sending it at all might be unconstitutional.
So, there would be challenges. I wouldn't say they couldn't do it, but I just don't think it'd be worth it. If there's a 20% gap or so between what would be accomplished via mandatory voting (using Brazil as a guide) and what we have now, I think there are better ways to close that gap. Moving election day to Sunday, making election day a national holiday, mailing out ballots to all voters, etc. would be better means to getting more votes imo.