What Went Wrong? [Archive] - Snowmobile World : Your #1 Snowmobile Forum

: What Went Wrong?


6
12-11-2002, 01:00 PM
Ok. I can see from the enormous amount of email and PM's that a lot of users want to know what happened to the forums on Tuesday night.

Here is a list of questions that users have been asking over and over again:

Q: What happened to the forums?

A: During a process to create a new area on SnowmobileWorld.com, we inadvertantly made a SMALL error in our database script (sql). When we ran the script to alter some tables in the database for forum integration in the new area, the error we made populated the post tables. The small error became much larger once the forums recounted it's statistics, as it no longer knew which posts came from who. As the forum software became more confused, it caused the database to become even more unstable, which kept snowballing from there. By the time the forum shutdown, the entire database was rendered useless.


Q: So how come it's running now if the database is gone?

A: We were able to restore the database from a 258 Meg backup that we have. Unfortunately, due to the size of the backup and the resources needed for a backup, we only dump the database once per week. Obviously, the last backup was done on December 5, 2002 at 10:00pm.


Q: So what happens to all the posts made between December 5th and Tuesday night?

A: We are sorry to say that they are gone for good. We have no way of restoring those posts.


Q: Maybe you guys should have done a backup before you started messing with the forums?

A: Good point. But since we were not actually trying to change the forum tables, we didn't feel it was necessary. Hindsight is 20/20 and we all learn from our mistakes.


Q: How long was the site down?

A: The forum software stopped all processes around 9:42pm Tuesday night. The forum, with great help from the mods came back online around 1:00AM Wednesday morning. Just over 3 hours.


Q: Why did it take so long to get it back up?

A: We needed to upload the offline backup of the database to the server, all 258 Megs of it. Then, the server needed to process the entire file and re-populate the database with nearly 80,000 posts. Add to that time to delete off any post from Dec 5th, 10:00pm til present.


Q: Was anything else affected?

A: We do not believe any other area was affected other than posts. PM's, new user signups, calendar events, etc., should not have been affected.


Q: Did a hacker do this to you guys?

A: No. We pretty much did this to ourselves and really don't need any more help messing things up since we have that already covered. :(


Q: Is there any way I can use this forum mess to confuse you into giving me a bunch more post counts than I actually deserve ;) ?

A: No.



We would sincerely like to apologize for any inconvenience this may have caused.

Thank you for your support.

MichYamaha
12-11-2002, 01:07 PM
One of the hazards in the networking world. I"m glad you were able to salvage as much as you did Corey, nice work :thumbsup:

got snow??
12-11-2002, 01:18 PM
very impressed with the minor amount of downtime you guy's and girls are on the ball around here and that is why this is the best snowmobile site period.... congrats and hope you good luck in the future. i had to talk to my wife last evening and thats as bad as it was around here. thanks

WoodyCam
12-11-2002, 02:05 PM
Good work and great save guys! I know because I run my own Dell Poweredge web server and understand all the fun when something like this happens. Keep up the great work and the great site!!

www.iwannasee.com

Joel

LB700
12-11-2002, 02:37 PM
Good work! One can only imagine the heartrate when things started going wrong on you!

One of the sites I used to visit (CMRC Forums -- now defunct for other reasons) got hacked, and our friends at www.ktmtalk.com helped immensely getting back online, albeit with loss of data.

If you need any assistance, you might want to contact them at admin@ktmtalk.com VERY helpful, and he specializes in data recovery.

snowjoe
12-11-2002, 02:42 PM
Corey,

As I said in one of the lost threads, No Problem !!!
Thanks for a great place to surf when I can't ride. SWRules

Joe

Steven Hohman
12-11-2002, 03:03 PM
Great job, Corey, I wouldn't have noticed anything had you not posted this....

Machzzzz1
12-11-2002, 03:13 PM
Computers are Junk. THey should be banned. :D

XCR583
12-11-2002, 04:11 PM
Great job Corey!! In the little bit of programming that I have done, I would never want to run a site like this. It was a fun night last night in the chat room while things were down. Some of you dont know what you are missing. It can be hilarious when there is a lot of people in there. SWRules

670HO
12-11-2002, 04:28 PM
Well good job of keeping the polyatomic krypton core from coming to a melt down.

dpaige
12-11-2002, 04:37 PM
Sounds technical???

What's it all mean.

LOL

nice job Corey, don't you just hate restoring from tape!!!

performancex
12-11-2002, 04:39 PM
I can't belive Corey retyped 80,000 posts in under 3 hours! Poor keyboard!

machz69
12-11-2002, 04:47 PM
not a problem guys and gals....hehehe and keep up the good work SWRules

cooley
12-11-2002, 04:53 PM
Originally posted by Machzzzz1@Dec 11 2002, 04:13 PM
Computers are Junk. THey should be banned. :D
hey machzzzz1 what would you do if they were banned and couldnt ride :doh: :doh:

performancex
12-11-2002, 04:55 PM
Without computers I could still ride.
I'd just be bored at work.

ZR Sled Head
12-11-2002, 08:16 PM
SWRules SWRules SWRules SWRules SWRules
Nice work and thanks for getting it up and running so fast.
:D :D :D :D :D :D :D

paul yarek
12-11-2002, 09:27 PM
how do them smut sites keep it together so well ?

Machzzzz1
12-11-2002, 09:28 PM
Good question.

konkinj
12-11-2002, 11:19 PM
Apology not accepted because you have nothing to apologize for. Your site is great and its free - Thanks - keep up the good work

mxz7
12-11-2002, 11:54 PM
Originally posted by paul yarek@Dec 11 2002, 08:27 PM
how do them smut sites keep it together so well ?
Ha Ha,from guys like you with credit cards :devil: :p

Wolfman
12-13-2002, 11:05 AM
Now I KNOW I've been too busy. I never knew the site had crashed. Glad to hear that all went well Corey. You guys do an excellent job in keeping us sled junkies sane!! SWRules

Skully
12-13-2002, 01:01 PM
Hey Corey A few suggestions for you.

Configure SQL to do hot database dumps to disk as often as possible based on you available disk space. Once per day is plenty for a forum. This way you won’t need to restore from tape unless your hardware craps out. A restore from disk with a database that’s only 300MB will only take 5 or 10 minutes. Backup the SQL dump to tape, not the actual database, daily or what every your hardware/software supports.

Although this is severely limited by resources, build a staging environment and test all changes against it before promoting to production. This will save you many problems down the line.

I do this type of thing for a living and support sites like City of Chicago, Figi’s, and Hub Group. If you want to talk in more detail send me an email and we can get into greater detail. If not, this is a great site and keep it up.

Skully
:thumbsup:

highlandergerman
12-13-2002, 11:00 PM
glad it is up an running, sometimes it is hard to just install a component, let alone run a website.