It could do. for instance say you installed some program onto your 2nd HDD that boots at start, it would still be loading from the slow hdd and that would slow down your system boot time.
I wouldn't be surprised actually.
What's strange is I can't seem to boot into the HDD, but I can still access it just fine and use it for storage. I might just transfer all those files to one of my externals and reformat it.