My son is further along than I, his recommendation to me (as we had this exact conversation last night) was to get 5 hearts at first so you can survive most one-hits and then about 1.75 stamina circles before refocus solely on hearts.
His statement was that once he got to that much stamina, he could climb / swim / fly as much as he ever has need to do without issue.