Outage explained

Information and news about the website will be posted here.
Locked
Andy
User avatar
Administrator
Posts: 12628
Joined: Fri Aug 18, 2006 11:47 am
Location: United Kingdom
Contact:

Outage explained

Post by Andy »

Hello all,

What a bloody night I've had...

9pm last night, C:\ ran out of space. I didn't understand why so I went to the temp folder and discovered a ton of files all around 170KB each, so I deleted them, but they kept coming back 1 or 2 a second.

I also found some odd looking files in the betaarchive folder. They weren't normal characters either, so I deleted them.

Come 10pm, the Temp folder was filling up again, so I did some investigation work with the help of DanielC and Mrpijey and we discovered it was a virus that had infected the system and was creating a load of temp files. It also infected all non-running EXE files, in both the OS and the website files, so anything that was an EXE has been deleted and will have to be restored from a non-infected backup.

It was nearly 1am before I even managed to backup what I could and do an OS reinstall. By this time I couldn't be bothered staying up to fix it, so I went to bed.

I came into work this morning an immediately started work on getting the server up. I've been on the phone to the DC for nearly 30 minutes, trying to get the NIC drivers to update etc, which was causing the main problem. It was only an hour or so ago that I managed to gain full control of the server with updated drivers and managed to get what I could back online.

No database data was lost, only exe files, which if running, were not affected (eg mysql, http, ftp, mail).

I sure hope this never happens again because this took the [censored]... seriously it did.

And thanks to the person who uploaded the file with the virus in. Yes it was a BETA, and I have a fair idea I know who it was. It could have been accidental or intentional but I can't prove either, so I'm going to forgive and forget this time.

Because of this I am now introducing the rule that ALL files MUST be RAR'ed or ZIPPED before being uploaded. No exe's or other extensions. This rule is final. Any exe's will be deleted with no questions asked. You HAVE been warned.

Problems aside, I hope everyone is glad the forum is back and that I never have to go through this again...

Enjoy the rest of your day

happy dude
Donator
Posts: 2461
Joined: Fri Oct 26, 2007 5:12 pm

Post by happy dude »

Ah, the joys of being a site admin.
Hopefully this doesnt happen again... but I guess some things are unavoidable.

Luckie
User avatar
Donator
Posts: 1115
Joined: Thu Aug 24, 2006 8:03 am
Location: Germany

Post by Luckie »

great, that BA is back online
"Theory is when you know something, but it doesn't work. Practice is when something works, but you don't know why. Programmers combine theory and practice: Nothing works and they don't know why."

viper
User avatar
VIP
Posts: 781
Joined: Fri Jan 04, 2008 9:18 pm

Post by viper »

Good work getting BA back online.

I am very glad its back up again

WeirdEars
User avatar
Donator
Posts: 192
Joined: Mon May 21, 2007 5:08 pm

Post by WeirdEars »

Were the files .TMP files and did they begin with 'POS' by any chance?

Ex. POSXXXX.TMP

Because my computer's been having exactly the same problem...The files came in quantities of 4,500 or so in both the C: drive and the 'My Documents' folder...

Andy
User avatar
Administrator
Posts: 12628
Joined: Fri Aug 18, 2006 11:47 am
Location: United Kingdom
Contact:

Post by Andy »

WeirdEars wrote:Were the files .TMP files and did they begin with 'POS' by any chance?

Ex. POSXXXX.TMP

Because my computer's been having exactly the same problem...The files came in quantities of 4,500 or so in both the C: drive and the 'My Documents' folder...
No, they were all random numbers and letters.

Gnome
User avatar
Staff
Posts: 2671
Joined: Thu Nov 30, 2006 6:10 pm
Location: Land of The Tea.
Contact:

Post by Gnome »

Yey,
ba back!
Well done Andy!!

RichardG867
FTP Access
Posts: 596
Joined: Tue Oct 23, 2007 11:21 pm

Post by RichardG867 »

Didn't noticed this outage (I study in the morning and I disconnected ~5pm GMT-3). (Edited 26/Nov/2009)

(Pointless part removed 26/Nov/2009)
Last edited by RichardG867 on Fri Nov 27, 2009 2:29 am, edited 1 time in total.

Vista Ultimate R2
User avatar
FTP Access
Posts: 2393
Joined: Wed Aug 30, 2006 10:06 pm

Post by Vista Ultimate R2 »

How did the virus in the uploaded file infect the server just out of interest, as it would surely have had to be run rather than just put on there?
Image

Gnome
User avatar
Staff
Posts: 2671
Joined: Thu Nov 30, 2006 6:10 pm
Location: Land of The Tea.
Contact:

Post by Gnome »

Any idea what it was called?

Pureelite
User avatar
Donator
Posts: 178
Joined: Sun Feb 03, 2008 10:49 pm
Location: Southampton, UK
Contact:

Post by Pureelite »

Just wanted to say thanks andy, i know you have been working hard at it!
Good work on getting it all back.

DanielC
User avatar
FTP Access
Posts: 156
Joined: Wed Nov 01, 2006 10:30 pm
Location: France
Contact:

Post by DanielC »

Toshua123 wrote:Any idea what it was called?
I found two when I scanned Andys backup ...

- Win32:Parite
- Win32:Parite-B@dll
~ Dan

Andy
User avatar
Administrator
Posts: 12628
Joined: Fri Aug 18, 2006 11:47 am
Location: United Kingdom
Contact:

Post by Andy »

DanielC wrote:
Toshua123 wrote:Any idea what it was called?
I found two when I scanned Andys backup ...

- Win32:Parite
- Win32:Parite-B@dll

What Dan said

I must have been infected by it when I "checked" one of the exe files was working, and it was infected. I was stupid enough not to have anti-virus because its so difficult to find a good one for server versions, and I had never had a problem in 3 years of running without one. Times change however, and when I get round to fixing the server back to 100% I will get round to installing an anti-virus package in the hope this never happens again.

stitch
Permanently Banned
Posts: 463
Joined: Mon Dec 11, 2006 3:09 am

Post by stitch »

I can donate a Symantec 9 license....

Bender
User avatar
Donator
Posts: 1524
Joined: Fri Aug 18, 2006 4:30 pm

Post by Bender »

lol

Code: Select all

- Win32:Parite 
- Win32:Parite-B@dll
is really old. I was infected with that in ~2002
Image
Mozilla/5.0 (Macintosh; U; PPC; en-US; mimic; rv:9.3.2) Clecko/20120101 Classilla/CFM
"Stupid can opener! You killed my father, and now you've come back for me!"

DanielC
User avatar
FTP Access
Posts: 156
Joined: Wed Nov 01, 2006 10:30 pm
Location: France
Contact:

Post by DanielC »

SP is now running antivirus, Andy, you have a PM so you can do the same.
~ Dan

SaT
FTP Access
Posts: 394
Joined: Wed Dec 27, 2006 5:44 pm
Location: Canada

Post by SaT »

why you dont install NOD32?
it will stop any type of virus

and you can use the trial version
Image

happy dude
Donator
Posts: 2461
Joined: Fri Oct 26, 2007 5:12 pm

Post by happy dude »

what emperium said...
Last edited by happy dude on Thu Feb 28, 2008 10:28 pm, edited 1 time in total.

empireum
Donator
Posts: 3557
Joined: Sat Sep 30, 2006 5:00 pm

Post by empireum »

happy dude wrote:*NO* anti-virus will stop ANY and every type of anti-virus.
Plus personally Ive never heard of Nod32 so I think theyll go wit hsomething more well known
That also depends if the site is on a Windows Server....
I think you mean "No anti-virus will stop any ... type of virus" Besides, NOD32 is a very good and thorough, yet fast scanner. It would also be my scanner of choice if I were running Windows boxen and were willing to pay for an anti-virus (as NOD32) is not free.

Vista Ultimate R2
User avatar
FTP Access
Posts: 2393
Joined: Wed Aug 30, 2006 10:06 pm

Post by Vista Ultimate R2 »

What I do is just have Kaspersky installed (got it legit now too, there was a promotion recently to get a free 1 year licence key) and scan anything I download from sources that can't be 100% trusted, I don't actually have it running in the background so I don't lose any performance to it - that would probably be the best solution on a server too.
Image

Locked