I went 2 and a half weeks last summer. The heat was intense but still I enjoyed every second of it. I was able to do Tokyo, Osaka, Kyoto (my fav), Kanazawa, Shirakawa, and Hakone before returning to Tokyo.
It was all beautiful, albeit maybe a tad condensed towards the end, but it was beautiful throughout. The group I went with continue to Okinawa so I would say 3 weeks is a great amount of time to go and let go! You can do quite a lot there, and I would recommend you put some time to explore the country side.
As long as you are organised and have a good schedule and itinerary, you can cover a whole lot in 3 weeks







