Restoring the lost data from the F4G crash.

Adam_UOForums · 17-07-09, 08:53 PM

Howdy all

Most of you probably remember the crash of F4G and the loss of data

Murad proposed restoring the data through a slow and somewhat painful way through the use of Google Cache.

(Original thread on here - Important.. yes I think so.)

He created a thread on UOforums about this matter a few weeks back stating that he managed to scavenge some 1.5GB of raw data, which MAY include the data that was lost a few years back. (See this thread for correspondance - Old f4g posts for UO guilds on europa - Ultimate Online Forums)

I've uploaded and imported the data on my own server to take a look and it's some 58,000 entries, roughly 21,000 of those unfortunatly are Archival duplicates

I've been trying to find a service, or person who would give me a quote on restoring said data into a readable format to either post as HTML or some other way

After receiving 3 quotes in excess of $1,000, I found someone who offered to do it for free and so far, the results look reasonably promising.

The updates I received in the last week or so were....

First update..

Quote:

I have been plugging away on this when I have a few moments here and there. I can reassemble the forums and subforum structure and built user ids - all with the same values as before. The script is doing a decent job on some sample data; maybe tonight I'll have some time to finish it off and run it against the database I pulled down from you.

Update TODAY was..

Quote:

Actually I have. The "rescue" code is more than 600 lines long now. I haven't been able to carve off any one big chunk of time to work on this but have been plugging away a little bit each day since; what I discovered is that the Google cache data is not all that consistent; there are plenty of dupes; some data has very little metadata in it to rebuild, etc. There still are 28000 pages and of course many more posts within those pages which I can recover complete with the original forum, thread and linkage info. Formatted as vbCode so the posts can be shoved back into a new forum, I wrote a little HTML to vbCode parser to do it that.

Left to do - sort out an issue with the members being parsed - for some reason I'm not finding very many valid members meaning a name and a member ID - that's sort of critical to rebuilding the complete post history. If nothing else I can insert an "apparently written by" text line for those posts, but I hope to do better.

Whatever the outcome I'd like to wrap this up today (meaning very late tonight my time - Pacific TZ) the code will produce sql insert statements for posts, forums, threads and members. Some assembly will be required. I'd recommend that an empty forum be created; these "rescue" records be added as new tables within the empty forum database, and then some selective sql updates be done to copy data from the rescue tables to the actual forum tables. That way I don't have to make the data look exactly like vB or whatever forum you choose.

So hang tight just a little longer.

Worst case scenario, we get back a readable format
Best case scenario means we are able to import it right into a forum.

The data of course, won't be imported into UOForums, as it's not mine, or my communities history here, it's yours.

I believe Moturdrn (sp) has received a copy of the raw data from Murad, I've also let him download it from the UOforums server to

I'll update if I get any news.

Irvyn · 17-07-09, 09:59 PM

Thank you

And to those who saved the data and are working on restoring it *hands large bunch of flowers*

Van Cocidius · 17-07-09, 10:32 PM

Technology man, it blows my mind. Thank you to all who've given the time. Very kind of you Adam to help out a "rival?" forum out like this, someone needs to give you a freaking medal. Fingers crossed it all works out.

Moturdrn · 18-07-09, 05:32 PM

many thanks for the update Adam :-)

Adam_UOForums · 19-07-09, 03:44 AM

Yea it's time consuming, to say the least
The good news is I've managed to import the forum names/posts/threads/members into a test forum, but it's linking them all together, which will be the difficult bit, if it's even possible, heh
There's only so much you can reimport with scraped data from google's cache
But if the posts/threads can be salvaged and reimported here, that'd be great.
Still messing with it at 4am, although it's something I've never worked with before, so it's a learning experience :P

After screening out all of the duplicates, I'm seeing the following raw data...

Posts - 173,142
Threads - 16,169

No doubt it's not all of the data which was lost, but if we can get back even some of it, it's a bonus.

Adam_UOForums · 19-07-09, 04:22 PM

Ok, so this is what I have so far, but I must warn you, it's not perfect, heh

VBTest - Powered by vBulletin

There are a few problems, to list a few that I know of, include..

1) Some posts are blank
2) Posts don't have a date/time of creation
3) Threads don't have a username showing in the forums
4) You can't search for threads created by a certain user.

With the majorly bad out of the way..

1) You can search users posts
2) You can search the forum itself (I believe, it works)
3) Posts seem to be relatively intact and most have usernames

I can provide the python script files used to extract the data from the raw html that was scraped from google by Murad
I can't provide the database itself, as it's a VBulletin product (I Believe it's illegal to share it? not sure)
But, what I can do is convert it into another (free) forum software like MyBB and then people can have copies of it freely
F4G admins of course, can have all the stuffs

Escaflowne/Drew · 19-07-09, 04:34 PM

Awesome stuff, nice one Adam!

Van Cocidius · 19-07-09, 05:24 PM

Yeah this is great! Thanks a bunch.

Sally Buttons · 19-07-09, 05:24 PM

Hurray!! Twistwam in Twinsic has been rescued! That was one of the most fun nights I ever had! Have a look.

Van Cocidius · 19-07-09, 05:37 PM

So will all this eventually be archived into fagnet after it's all been debugged or such, or would it be worth me copying each thread myself (Trinsic only) into an archived area for our guild to gander at?

Adam_UOForums · 19-07-09, 05:48 PM

It's upto the F4G admins if they want the data
I know Motrudan has the raw copy, before it was stripped of junk, but I can provide him or the owner the stripped copy to do with what they want.

Adam_UOForums · 19-07-09, 08:28 PM

Importing it all into the freebie MyBB, seems to have gotten a better result, ironically

F4G Net Restored Data

I've managed to remove all the forums with no data
I've restored the proper post/thread count as well as the usernames who created the threads etc.

If anyone would like a copy of this intact database, I can provide it
You'll need an install of MyBB to make it work like I have.