I think it's a little more than just memory. Here's a visual of each ship just prior to their arrivals on the planets:
Similar in many ways, they share the same shape, and they both have the Deus weapon at the core.
In Xenogears - the party explores the ruins of the ship in two segments, first - the tower of Babel which encompassed much of the living space. It housed enough population that they required a monorail to get around on the ship: so probably in the millions. The portion that lies beneath the ocean, where the weapon is housed that had been loaded aboard the ship. The ship is very clearly the ruins of what was shown in the opening cutscene, a colony ship. So I think we can accept that as not a bad memory. We also see a cutscene where Abel makes contact, but this occurs after the ship had crashed. In addition to Fei's memory of Abel making contact with the Zohar, and the formation of the original Elly, The Wave Existence also confirms the story, and confirms that Abel made contact after the descent to the planet. Elly was formed due to Abel's loneliness.
In Xenosaga - the ship is completely devoid of any living quarters and doesn't even seem to have much or anything in the way of technology aside from the ES units (and Yuriev's that looks like the Xenogear). It's a large vessel that looks like the internals of giant Gnosis; it's like internal guts and energy waves.
The stories we see of how the ships arrive are different too.
In Xenogears - we see a colony ship traveling through space. The Deus weapon reawakens, and the crew attempt to stop it. The captain self-destructs the ship, and it crashlands on an alien planet. Abel is the only survivor, and he makes contact with the Zohar after the descent (as mentioned above).
In Xenosaga - the ship explodes at the end we see Nephilim transform into Elly and, along with Chaos and Abel, teleport the core of the ship to Lost Jerusalem (Earth).
So while I think Xenogears and Xenosaga are following the same general plotline, they are two different stories. Like a retelling of the Xenogears story rather than them actually being in the same universe. Kind of like how Disney's Hercules use the same character as Hercules the Legendary Journies, but they are not really the same universe. I think Takahashi wanted to tell a story from that era, but didn't want to be bogged down and limited by having to remain consistent with Xenogears. Also, Square owns Xenogears; although - I will note, that is not a limitation on them connecting the stories now, with a bit of Nintendo influence - and maybe money - it would be trivial for them to arrange to license anything in either Xenogears or Xenosaga. Essentially, if Nintendo wanted to make/remake a Xenosaga or Xenogears game, they could easily arrange to do it.
Yeah, essentially we are looking at episodes 1: interplanetary war and man's scattering to the stars (also the creation period for Deus a biological and mechanical being, 2: Eldridge's fall to Earth and time of Abel, 5: Resurrection of Deus and Fei's awakening.
While some things in Saga and Blade X have changed since perfect works, there's still a definite connection. Intergalactic war, Earth's "destruction", the Galactic Federation, "Neo Jerusalem" and "New Los Angeles" (or New City of Angels). Certain tech from Xenoblade X could surely be seen as prototypes for tech that appear in both Gears and Saga.