Athiests don't have a dilemma because we don't make any claim. We may not know why the BigBang/Universe started because there is no justified explanation for why it started. With all the data we have currently, we just don't know or understand. We are not forced to come to any conclusion. It is okay to say we don't know. There may be a reason why everything started from nothing and of course we don't understand it currently. But just because we don't understand it doesn't mean we should just blame it on God. In cases like this, we just say "we don't know".
But of course, some athiests are "strong atheists" who do believe that there are no Gods.
@Runa. I'm gonna continue with Jay if you don't mind, try to see if my answer to him better answers it for you too.
So Jay, there is a scenario which requires an explanation, to which the current answer is "we don't know" right? In essence the question of, what is the origin of the big bang.
Currently the answer is we don't know, just like for the believer the answer about how God is the origin of all things material is "we don't know". However, deists believe God is the end of the chain of questions.
For an atheist, is there an end to the chain of questions, or do we just not know? Also, if there is an end to the chain of questions, would it be a fully naturalistic explanation, completely self sustaining? Something like, in exceptional cases, matter and energy can be spontaneously created out of nothing, for absolutely no reason?