Print at Jun 29, 2022, 10:51:51 PM
Posts: 22   Pages: 3   [ 1 2 3 | Next Page ]
View all posts in this thread on one page
Posted by mazoola at Nov 18, 2015, 12:26:59 AM
Suddenly, huge file size...
Today, I noticed sh3d seemed to be taking an awfully long time to save my current home plan -- and when I checked my data directory, I discovered the file was at 660 Mbytes and counting. (For the past month or so, the plan typically hovered around 145 - 170 Mb; my most recent offline backup -- from 4 November, alas -- was around 70 Mb, compressed.) Even though the file still appaeared to load OK, I was reluctant to continue using it, as I feared it would eventually fail completely.

When I saved the .sh3d file compressed and opened it using an archive client, I discovered two oof the entries weren't as one would expect: Items 300 and 316 in the ContentDigests were both listed as "###/123/untitled/texture_3.jpg" -- but the 300 and 316 folders actually held seemingly complete copies of the entire plan. (More correctly, they appeared to hold complete copies of slightly earlier versions of the plan.) And in case that wasn't enough, inside folder 300 there was another copy of folder 316 (numbered '315' in this instance) for a third level of recursion. Folder 300 weighed in at 314 Mb, with 316 contributing another 156 Mb.

I tried hand-editing the compressed plan file, but could never get it clean enough for sh3d to accept it. (I'd get the erroneous file warning asking me to choose whether I wanted to replace or remove the missing objects; no matter what I chose, though, sh3d would quit, leaving me to kill the java process.)

In case the problem was caused by a corrupted model, I went through and deleted everything I'd added in the past two weeks, resaving after each deletion -- but to no avail. Since the failure didn't seem to be object-linked, I decided to take it the other direction -- starting with my two-week-old backup plan, I cut-and-pasted only the changes I'd made from the corrupted plan to the original. Unfortunately, when I went to save the updated plan, it again ran greater than 650 Mb. (I later discovered copying the doors and windows from one floor of the corrupted file to the back-up would cause the expansion -- but deleting them afterwards only brought the file down from 650+ to 335 Mb.)

Finally I returned to the backup plan and added all of my recent changes from scratch, re-importing the models. When I was done, the uncompressed .sh3d file was 172 Mb.

I also noticed there were two 'UNRECOVERABLE' files in the sh3d 'recovery' folder under 'Application Data' -- one 120 Mb and the other 327 Mb. These are near enough the sizes of folders 300 and 316 to suggest some link -- especially as this plan has never legitimately hit 200 Mb, let alone 300. Deleting those files, though, seemingly had no effect.

I understand this was, more than likely, a purely happenstance occurrence; given I have no idea what caused it -- or, for that matter, at what point in the past two weeks it happened -- I realize there's virtually no chance of reproducing it. I did want to pass along what little I was able to find, though, on the off chance it might help you resolve a known issue.


Posted by Puybaret at Nov 18, 2015, 9:35:36 AM
Re: Suddenly, huge file size...
Sorry for this problem. It's not the first I see it and wonder how it can happen. A copy/paste operation between files could be a possible track, but I didn't succeed to reproduce this "recursion" issue after new attempts.

For your information,
- The recovery folder is the place where a copy of the edited home is automatically saved in a file with a .recovered extension, each time the delay set in Save home data for recovery every preference expires. When you quit normally Sweet Home 3D, the .recovered files in that folder are deleted. So these files are kept only if the program crashes.
Each time the program is launched, it reads and opens the .recovered files in that folder to let the user recover his recent changes after a crash. But sometimes the automatically saved file is incorrect and it can't be reopened. To avoid trying to open a wrong .recovered file at each launch, it's renamed with the .unrecoverable extension. There are very few changes that this file can be used, but who knows?
- The ContentDigests entry in a .sh3d file gives the SHA-1-Digest value of every entry of the .sh3d file used by Sweet Home 3D. This value is used to quickly check whether an entry is correct or not. So if you edit an entry in the .sh3d zipped file, Sweet Home 3D will think the file isn't correct anymore. I didn't program this feature to avoid users to manually edit an .sh3d file of course, but to check if an error on the disk didn't happen and inform the user accordingly. If you want to bypass this data integrity check, just remove the ContentDigests entry of a .sh3d file before opening it with Sweet Home 3D.

As you can see, I pay a lot of attention to users .sh3d files. wink
Emmanuel Puybaret, Sweet Home 3D developer

Posted by mazoola at Nov 19, 2015, 6:59:23 PM
Re: Suddenly, huge file size...
If you want to bypass this data integrity check, just remove the ContentDigests entry of a .sh3d file before opening it with Sweet Home 3D.

Good to know! When I get back home, I'll see if I can correct the corrupted file this way. (I should also metion I'm running 25? 26? 27? levels in this plan... wink )

Posted by mazoola at Nov 25, 2015, 6:33:18 AM
Re: Suddenly, huge file size...
I returned home late last night, and one of the first things on my list for the morning was to see if I could simply delete the corrupted directories and the ContentDigests file from the .sh3d archive and have the plan load correctly. First, though, I wanted to check out a few models I'd built while I was away. (In advance of some podiatric surgery, I'd been helping a friend reorganize her home so all critical living functions would now take place on the ground level. In the evenings, I'd occasionally commandeer one of her PCs and refine some Sketchup modeling on which I've been working...)

The new designs appeared to work well, and after a couple of hours spent tweaking colors and textures, I thought I had the items reasonably well integrated into the home design. I decided I'd break for lunch and tackle the question of hand-editing .sh3d files afterwards. Before I stepped away from the keyboard, though, I hit 'save'...

...which suddenly seemed to be taking an awfully long time to handle the request. Sure enough, when the app finally returned control, my 78 Mb (compressed) plan had suddenly grown to 373 Mb (uncompressed). Inside the .sw3d file, I found item 315 now pointed to a folder containing an almost-complete copy of the plan, weighing in at 195 Mb.

Based on what I hope was an accurate reading of your previous post, I removed the offending file from the zipped plan, along with the entire ContentDigests. As expected, when I attempted to load the edited plan, sh3d responded with a warning the data file was invalid; while I could continue to load the plan, one of its furniture items would have to be removed or replaced with a placeholder object.

Unfortunately, as was the case last week as well, regardless of whether I choose to remove or replace the offending object, sh3d hangs. Last week I thought the process suspended and had to be cleared manually. This week, though, I saw it continued to perform at least some calculations, although the process's CPU utilization dropped to below 0.20%. I couldn't determine if the threads being executed were typical or correct for the application at that stage of invocation, or if they indicated the system had entered into an irrecoverable loop. In any case *I* didn't have the patience to wait a few hundred minutes to see if the system would eventually give me a welcome screen.

Once again, I couldn't identify the moment the corruption took place;. nor could I recall any unusual action I'd taken that could have introduced the error. I'm disappointed I still can't get the edited plan to load, as it means I most likely will have to hand-assemble a replacement plan by taking my most recent known-good file -- which is now three weeks' old -- and manually repeating each change since made. (Obviously, such an approach can only continue to work for me for a limited time.)

Again, any pointers or corrections most gratefully accepted!

Posted by HawkDawg at Nov 25, 2015, 6:44:02 AM
Re: Suddenly, huge file size...
I'm having the same thing happen to me. I'd be interested in why SH3D creates this duplicate folder containing all other duplicated folders and files inside of it, and why - when deleted - SH3D won't load.

Edit 1: Puybaret - I can make my 70 MB file available on one of my websites if you'd like to have a look at it.
The folder with more folders inside is about 25 MB.
I zipped it up using 7Zip. As a zip file it's about 30.5 MB. As a 7z file it's about 19 MB.

Just let me know if you'd like to take a look at it, to try and figure out what's going on, and I'll upload it and give you a link.

Posted by HawkDawg at Nov 25, 2015, 8:37:43 AM
Re: Suddenly, huge file size...
A copy/paste operation between files could be a possible track,

I think this might be what is creating the folder with the sub-folders and files.
I looked through mine and it looks like it is filled with models I've copied/pasted within the plan, although I didn't look through every folder.

Posted by mazoola at Nov 28, 2015, 5:14:20 AM
Re: Suddenly, huge file size...
OK, it's not solely linked to copy/paste between files.

The past few days, I've been compressing and checking each save, being careful not to copy/paste from file to file -- and until a few moments ago, it looked as if that might be the answer. Unfortunately, I just saved a plan that should have weighed in around 190 Mb uncompressed, and it ended up over twice that size. Compressing and opening the zipped file shows it contains a recursively saved plan.

The only oddity I noticed was that during the save process the UI seemingly "jumped" -- shifted over a few pixels , with the scroll bar on the object pane disappearing. Whether that's of any help, I don't know....

I have a suspicion or two to investigate, but wanted to pass along the little I've learned.


Posted by HawkDawg at Nov 28, 2015, 6:28:43 AM
Re: Suddenly, huge file size...
It will be interesting to see what your investigation turns up.

Posted by mazoola at Nov 29, 2015, 9:54:53 PM
Re: Suddenly, huge file size...
I'd worked for most of a day with no problems, with the plan working its way to about 200 Mb. (I added several complex models built in Sketchup, so the sudden 50 - 60 Mb jump is normal.) However, after making a few tweaks to fix corrections I'd subsequently lost when reverting to an earlier save, I hit 'save' one last time -- and ended up with a 400 Mb file. (Fortunately, I actually hit 'save as,' so I only lost my final half-hour of fiddling about.)

Once again, I *didn't* copy/paste from file to file; however, I *did* perform a number of copy/paste operations from level to level.

I also deleted several [empty] levels I'd mistakenly added by hitting the '+' tab instead of my target.

In addition, I copy/pasted several *walls* -- not just furniture -- from one level to another. (For each of the 6 floors in my plan, I'm using as many as 7 co-located levels, for a current total of 24? 25? levels.)

I also experienced a previously unseen glitch earlier, when, after modifying an object's materials, its normal onscreen display was replaced by a featureless box with a texture applied. (As the texture happened to be the JPEG used as a diffusion map for the standard mirror object, after I'd edited a handful of objects, my plan looked like a mid-1990s showcase kitchen: shiny stainless steel as far as one could see.) Objects rendered without a hitch; only their 2D representations were corrupted.

I don't know if this suggests the recursive bug had occurred before my actual attempt to save, but while the program was laboriously writing its 400 Mb plan to disk, I took a look at sh3d's working directories. Inside the contemporary 'work' folder, I discovered two [auto?] saved plans, *each* weighing in at 400+ Mb. (Unfortunately, I failed to make a note of their respective timestamps, but IIRC, at the time of the save, they were both a dozen or more minutes old.) During the actual save, it appeared that both my .sh3d data file and the .RECOVERY file were being incrementally saved in tandem. Once the save was complete and I exited the program, both the .RECOVERY file and the current work folder were deleted automatically. (However, I still have four work folders dating from the 24th and 27th, totaling 1.1 Gb. I assume these are the result of a program crash -- quite possibly during my unsuccessful attempts to load a .sh3d file from which I'd removed a recursively saved plan -- and can safely be deleted.)

Unsurprisingly, every suspicion I had (primarily involving multitasking during the save process) has proven unfounded, and the problem occurs *just* infrequently enough that when it does, I typically can't reconstruct exactly what it was I'd done since the last problem-free save. When I return to sh3d later tonight or tomorrow, though, I'm planning to log each action, in hopes of nailing this sucker.


Posted by HawkDawg at Nov 29, 2015, 11:03:15 PM
Re: Suddenly, huge file size...
This is of no help to this particular issue but I thought I'd mention that all folders and files in the work folder can be deleted if they start getting to be too big, with no negative issues, as per Puybaret stating so in this post:,5683#24718

Posts: 22   Pages: 3   [ 1 2 3 | Next Page ]