Thanks to the support of all the creators who were impacted, 459/519 files have been restored so far. We’re responding to creator and customer emails as soon as we’re confident everything is back where it should be, so if you’ve contacted us over the past few days, you should be hearing from us soon.
We’ve verified the object store file locks are functional and a full backup is currently in process.
We’re currently have problems serving some files.
When the file is downloaded, you may see an error message of “NoSuchKey” or “The specified key does not exist”.
This is due to an unintended side-effect of a process we undertook to manage thumbnails (!). See below for more technical details about this.
We’ve identified the loss of 519 individual files over 265 products (out of more than 16,000 files over more than 10,000 products). These products have been hidden, to ensure new buyers are not caught unawares.
We’re working with creators whose products have been affected, to restore access as fast as we can. If we’re unable to restore lost files in a timely manner, we will be in contact directly with any customers who are impacted.
We’re deeply sorry for the inconvenience caused to both creators and customers, and want to thank you all so much for bearing with us as we sort this out.
Piete & Steve.
MH & WG3D are running on the same server. Due to the way thumbnail generation was configured for both sites, the disk space was starting to get a bit tight.
On Friday 25th November, we ran a cleanup process on extraneous thumbnails for products updated during November. While the models are stored separately on different object stores, there is some synchronisation between the main server and the object stores.
We believe a combination of this cleanup process and the synchronisation had the unintended side effect of removing a number of uploaded models as well.
There are some controls available on the object stores to lock files, however these were also disabled.
We’ve contacted the vendors of the impacted products with all the information needed to help restore normal service.
We have enabled a number of object store service controls, aimed at making it harder for files to be deleted.
Additionally, we have started work on an incremental model backup process so if the worst does happen, we have a better way of recovering.