Site Discussion

What the hell happened to all those hundreds of images: A nice, long technical explanation

posts: 645
Ugh, right, so the software that the TakaWiki runs on is called tikiwiki, and every six months or so they release an update with security patches, etc.If we don't keep up with the updates we become vulnerable to attacks, and if we don't keep up on a semi-regular basis, our software falls so far behind and becomes so incompatible that an update becomes a nightmare (remember when all our Japanese became corrupted and we had to manually fix EVERY. SINGLE. PAGE? Yeah, that was after we hadn't updated in five or so years).

So, to do an update we do a clean re-install, which means deleting all of the files related to the old version, then installing all the files for the new version. This is a pain in the ass and takes about 5 hours. But if we don't do it, again, corrupted files and other kinds of nightmares I don't want to imagine. So, we do this every six months or so and every time I hold my breath and cross my fingers (see upgrade nightmare above).

We are a database, and the mysql database stays on the server, not being changed until after the upgrade is complete and we actually instal the newest tikiwiki software. Just in case the upgrade self-implodes, I backup the database before every upgrade.

Also, there is a folder in amongst all the software folders, where images used to be stored. As in, prior to 2014, the tikiwiki versions stored all the images as nice jpg or png files there. So after every upgrade, just drag and drop the folder with the images back into the new version's folder on the server et voila, images back.

Well, then tikiwiki got fancier. After the 2014 upgrade (remember when suddenly adding images included about ten more, really annoying steps? That was the upgrade) suddenly it began storing images in galleries when you uploaded them, and it gave you a choice: Store the images as actual data in the database, or as files in a folder. Well, at first I thought, ah, in the database! Then we won't have to drag and drop images with every upgrade, they'll all be there in the database.

So we chugged along like that for several years, and then I started reading advice from people who said "If you have more than a hundred or so images, you shouldn't store them in the database because it slows it down." Well, guess how many images we had stored by that time? About 2700. (This is not counting the images prior to 2014, just the ones after that were stored in the database.) Well, this did not sound good at all, because we had a LOT of files and I knew we'd only be getting more. So I switched it over to storing as files in a folder, peeked at the image folder on the server, and went "What the hell?" Because these new image files were not nice, clean jpg and png files. They were strings of code like this "0ae5a1a6e60e71006982d83ca77b79dd" completely lacking file extensions and unable to open with any kind of normal image viewer. Still, they were in the folder and everything seemed to be working.

Cue our latest upgrade, where I did my usual backup of the image file folder and dropped it back in after the upgrade. And cue the absolute disaster. Because what seems like half those images were completely corrupted (either not displaying at all, or displaying weird gobbledygook). And when I went to try and individually fix them, of course those weird extensionless files which I had backed up were completely useless.

So, the moral of this story is that I am going into the TakaWiki and manually pulling out all the images which survived intact and saving them as jpg or png files, and I will do this going forward as well, so we actually have a safe backup in case this nonsense happens next time. And I will be going to the tikiwiki boards to see what the coders think went wrong.

So, in short, I apologize for all the files from 2014 to now that are gone. I know people put a lot of time and effort into adding images. :( nd I'll be doing my damndest to see this doesn't happen again next upgrade.

posts: 645
Thanks. :)
posts: 7 Costa Rica
OMG! Jen I sorry
posts: 645
Ugh, I think a lot of the problems is that I don't know much coding or software development stuff at all myself, so every time something breaks I have to teach myself just enough to fix it. .... And who knows if I'm not just causing problems for myself down the line.

Still, free software, hard to complain too much. ;)
posts: 26
Holy smokes how unfortunate! Hearing about how this system appears to be fundamentally structured, this sort of incident sounds inevitable but absolutely unacceptable. As a tech person myself who regularly harrases my team of users for feedback so they never find themselves in a similar bind, I really hope those devs respond with a strong solution.

Thankfully pictures of siennes / posters / etc are fun to look at lol, so it'll all be back soon!