Expansion Storage Prices Are Out of Control

[Zkuq]

Pemalite said:

Zkuq said:

Really? Do games duplicate data because of optical media despite probably everything getting installed on mass storage anyway since like ten years ago...? I thought it had something to do with seek times, which, as far as I know, is an issue with HDDs as well.

Actually a quick search online reinforces my understanding, but I didn't look into it that deeply.

On optical media the game developer has the power to determine what data can be duplicated and where.

On a consoles internal storage, game developers don't have any control to dictate what part of a game gets installed on what sectors of a mechanical hard drive... They cannot shift the install of another game while they are installing to be on certain parts of the drive.

The performance of a mechanical hard drive is still orders of magnitude better than optical media.

Zippy6 said:

Asset duplication was absolutely a thing on HDDs, hell it's likely still done on some games now as it's just how developers have structured their files for ages, but it was more a necessity on HDDs for streaming assets.

Mark Cerny talked about it before.

"Without duplication, drive performance drops through the floor - a target 50MB/s to 100MB/s of data throughput collapsed to just 8MB/s in one game example Cerny looked at. Duplication massively increases throughput, but of course, it also means a lot of wasted space on the drive."

https://www.digitalfoundry.net/articles/digitalfoundry-2020-playstation-5-the-mark-cerny-tech-deep-dive

Spiderman for example stores all the assets needed for a city block together, so when that block is loaded as you move through the city all the data is together on the drive to be read fast. Which lead to them at one point having a trash bag asset duplicated over 600 times and taking up a gig of data before they kept it stored in RAM instead.

A lot of games will have big chunk files for levels that contain all the assets for that level, even if the asset has appeared in another level previously.

A game developer doesn't get to choose what chunk of sectors they will occupy in a file system.. So they can never guarantee a block of data will exist next to each other, they cannot demand data be evicted from one area and moved to another for performance reasons.

The seek times from the inner most tracks to the outermost tracks of a hard drive is around 10ms.
And if your hard drive has latency of 250ms... Then something is wrong with your drive, which makes me question that article, that is optical drive latency figures.

Now consoles like the Xbox 360 leveraged XTAF in order to *try* and ensure data is available sequentially, Xbox One moved over to exFat which relied on indexing to keep track of chunks of data spread across the entire drive... And because of that, fragmentation of data is a massive issue on Xbox One which can result in performance degradation over time.

If you put related data contiguously, I bet you the chances are it'll be together on the disk as well. The file system is free to do whatever it wants, but most sensible file systems probably try to not scatter data around when they know it belongs together (e.g. if it's a single file, even if that single file comprises of a large number of individual assets). That to me seems way different than e.g. storing each asset as an indivual file, which the file system is much more likely to scatter around because it doesn't know they're related - especially if they're in different directories and could be used in a number of different situations. If you need, say, 2,000 assets for a scene (often a map), and it takes 10 ms to load each one, that's already 20 s in total just for seeking, let alone actually loading the assets. I don't know how many assets you need to load at once in a modern game, but I don't think 2,000 sounds way off for an AAA game. Regardless, even if it's just 200 assets, it's still 20 s just for seeking, then some for loading, some for actually running code etc., and I don't think it's hard to see why loading times might be high if data is scattered all over the place.

Developers do all kinds of stuff that's not guaranteed to work, and I'm sure game developers do it just as much if not more than others. I know that in low-level programming languages it's common to consider how exactly data is aligned in memory, how the CPU is likely to cache it etc. even though there might not be any actual guarantees. If it works now and has seemingly little reason to ever change on the target platform, why worry about it not being guaranteed? Stuff like that makes me uneasy personally, because I like guarantees, but that's also something that happens and works to my best knowledge.