Update on the Missing Forums/Posts

Last Wednesday (4/27) the Community Interaction category of the forum disappeared. I know many of you have been wondering what happened, what is currently going on, and when it will be back. This post is to explain the situation - and does so in detail - so as to be transparent about the problem, since it's still unresolved. It may get somewhat technical. If you want the short explanation, jump to the end. I would have made an announcement sooner, but I had to gather information and I had expected it to be resolved by now.

What Happened?

On 4/27 I was updating forum software add-ons by deleting old add-ons and updating existing add-ons to their latest version. Add-ons provide the forum additional functionality, such as our Resources, Medals, Blogs, FAQ, Gallery, and others. My investigation suggests that this process triggered the deletion of the Community Interaction forum category and it's child forums.

In my investigation process I've shut down the forum a few times to take database backups. Fixing this entire issue had become more complicated, and thus taking longer, because when I attempted to work on the database I encountered technical server-sided issues that couldn't be explained. This prevented me from accessing or backing up the database.

After working with our web host, they suggested the data directory was damaged. The data directory stores database information for all databases on a server, not a specific site's content, and allows databases to operate (a database for databases, if you will). We spent three days mapping out a technical plan to fix the issue with the data directory (taking backups, resizing the server's disk space, etc). At the last minute, the server's support team told me the issue had been resolved without the steps we'd been planning. This was because the cause wasn't corruption/damage of the data directory, but that the password contained a character the server could not read, and thus threw the error. This put restoration process of the Community Interaction category back several days, but tells me that the issue was probably not caused by the issue on the server.

In short: the category and forums are missing because a software add-on removed them unexpectedly. Resolution of this issue has taken longer because of an significant unrelated issue with the web host, which has now been resolved.

What Can We Do?

In examining this issue, I've also spoken with Xenforo support staff. They strongly suggest restoring the forum from a database backup. The problem here is it would remove all posts made after the backup, and our most recent backup was from 4/19 when we upgraded to the latest version of Xenforo 1.4.x.

Not wanting to pursue this solution, I found a similar situation on a different forum that was able to restore forums and posts that went missing by repairing the database. This only works if the cause was triggered by the software add-ons as suspected above, that the database queries are implemented correctly, and that the information wasn't deleted from the database in the process.

Another option was to merge the old backup with the current version of the site, which was discouraged by Xenforo Staff. This is because it's a very technical process (you need an expert) and you risk causing other issues in the future (damaged functionality, incompatibility issues, inconsistent data, etc.).

This brought the options to the following: 1) restore the database from a backup and lose data since 4/19, 2) confirm the data still exists and repair the database, 3) merge the old and current database and risk peripheral issues (serious technical expertise required).

Option #1 is the least appealing to me, and becomes less appealing every day this isn't resolved, because we'd lose more information. Option #3 risks unknown future issues and requires an database professional (expert). Fortunately, I found a professional who is willing to help with this process, but even he does not like the merge idea. He's willing to attempt it but has not been available the past few days.

Option #2 is the best solution, but may not work. It requires some level of technical expertise to run the queries and make sure everything is how it should be. I've consulted with a few people and they've outlined possible queries that should fix it if we're right about the cause.

What We're Doing

After fixing the second issue with the server's data directory, I've been preparing the server to run a private test installation of the site. This process basically includes taking backups of other database on the server and deleting old content from the server to make room for the test site. Our forum database is huge, and approaches 30 gigabytes, so we need more space.

I believe I now have enough disk space to set up a test installation.

What I am doing today is setting up a test installation of the site based on a backup I took two days ago. After successfully duplicating the database, I'm going to make some adjustments in the database and then run the queries given to me by someone who's had a similar issue happen on their forum. Essentially, if the data wasn't deleted from the database, it's still there, it just isn't assigned to a forum or category - it's orphaned. These queries should reassigned the orphaned threads to a new forum/category, which should restore the missing content.

I've taken an early look and we have a huge number of orphaned threads. Our database size now is also approximately the same size as the backup I took two weeks, which indicates our data is safe in the database, but simply has to be repaired.

I will run this test repair today. If I'm successfully, I will consult with the database professional and have him confirm the process. We will the implement the same process on the live installation of the site.

If we're not successful, the database professional has agreed to attempt Option #3, merging the two databases.

Final Thoughts

I apologize for being so detailed and technical about the process, but I want to be transparent about the process because it's been long enough that I owe you an explanation, and because it is a serious issue.

It's complicated and unfortunate. This post isn't intended to freak anyone out about losing content, and I realize that if we lost the entire Community Interaction category it'd be devastating. I don't think this will happen. I'm confident this problem will be fixed and will be by the end of this week (hopefully sooner). Please be patient. I'll make updates as the process continues.

After this is behind us, I am making some changes to prevent anything like this happening again in the future:

• Scheduling automatic daily database backups
• Testing changes to the software on a test installation before a live update
• Updating our server to the latest hardware and software
• Implementing additional security protocols to protect the site from other risks, such as spam and hacking
• Optimizing the forum's technical operation so it runs faster and more efficiently

TL;DR

An upgrade to a small part of the forum's software may have deleted or hidden the Community Interaction category. There were other issues that caused a delay in a fix. This is a serious issue with some risk but it looks like we can fix it. Options include restoring from an outdated backup, finding a way to repair the database, or merging the backup with the current site (all have benefits/risks). Today, I'm testing a fix on a test installation of the website and I am consulting with a database professional who can help us repair and restore the data. I expect this to be solved earlier than the end of the week. We're taking steps to prevent anything like this happening again in the future.


Thank you for your patience!

- @Daniel