The server is held together with cron jobs!

Information and news about the website will be posted here.
Post Reply
Andy
User avatar
Administrator
Posts: 12629
Joined: Fri Aug 18, 2006 11:47 am
Location: United Kingdom
Contact:

The server is held together with cron jobs!

Post by Andy »

Well I said I'd get round to giving you guys the lowdown on this one, so here it is!

As you know the server does a fair amount of work on its own. It went 2 weeks without me even touching it while I was in hospital, something not many servers can do. The reason behind this is the cron jobs running behind the scenes keeping things running and fixing potential problems before they occur.

I have a number of cron jobs running at various times.

1 minute cron job
10 minute cron job
1 hour cron job
4 hour cron job
1 week cron job
So, what runs on each one? Most are self explanatory but I've put some comments in green for those that aren't so obvious.
1 minute cron job
The FTP statistics page update script
The mini-status update script
The memory usage update script
The CPU usage update script
The UPTIME update script
MRTG Traffic Graphs update script
10 minute cron job
Setting the CPU usage priority of the following processes to LOW:
PRTG Traffic Grapher.exe
DFRGNTFS.exe //Defrag, which doesn't get run often
WinSCP.exe
IRCXPro.exe
FileZilla.exe
uTorrent.exe
FileZilla Server.exe
php.exe // PHP4
php-cgi.exe //PHP5
sven1.exe // Game server process
sven2.exe // Game server process
gmod1.exe // Game server process
css1.exe // Game server process
Delete all files in d:\*.tmp // temporary files build up due to a php bug, so this deletes them
Delete all files in d:\public_html\betaarchive\*.tmp // temporary files build up due to a php bug, so this deletes them
Run the awshost status update script
Run the sessionfix script for the forum // without this the forum kept getting session errors, so this runs every 10 minutes regardless of any errors
Kill all cmd.exe processes running
1 hour cron job
Kill all php.exe processes // php sometimes crashes, so this kills all the processes once an hour in case this happens, so the max downtime is 1 hour
4 hour cron job
Kill all DeltaC.exe processes // this is the backups, which have a habit of crashing, so they get killed every 4 hours and reopened by the next cron job
Kill all rsync.exe processes // this is the backups, which have a habit of crashing, so they get killed every 4 hours and reopened by the next cron job
1 week cron job
STOP and START FileZilla Server FTP Server service // Filezilla drops connections when open a long time so this restarts the service once a week
Kill all netgraph.exe processes // when left open netgraph uses a lot of memory so once a week its killed
START netgraph.exe // this re-opens netgraph after being killed above
Most of the kill and start crons are designed to simply kill all unused processes that are left open by some operations, taking up memory. In the case of netgraph, it is killed and reopened once a week to clear old data from memory, as it doesn't do this itself. Over the course of a few weeks it takes up a few hundred MB of RAM. FileZilla Server seems to have issues keeping connections if open for weeks at a time, so a quick service restart of that, taking just a few seconds, solves this issue. It does mean anyone who is connected gets disconnected but its once a week so it doesn't really affect anyone that much.

So there you have it, thats how the server seems to fix itself!

happy dude
Donator
Posts: 2461
Joined: Fri Oct 26, 2007 5:12 pm

Re: The server is held together with cron jobs!

Post by happy dude »

For those who don't like reading, BetaArchive would simply like to advise its users that the server in its current state is held together with a combination of hot glue, masking tape and carefully placed rubber bands, so carefully placed that if even one were to be pulled or removed the whole server would disintegrate

and all that [censored]

Andy
User avatar
Administrator
Posts: 12629
Joined: Fri Aug 18, 2006 11:47 am
Location: United Kingdom
Contact:

Re: The server is held together with cron jobs!

Post by Andy »

I've never heard it put like that before, but I suppose thats about right!!!

happy dude
Donator
Posts: 2461
Joined: Fri Oct 26, 2007 5:12 pm

Re: The server is held together with cron jobs!

Post by happy dude »

This is why OVH don't release pictures of their datacentres. You'd be horrified if you saw the physical state the servers in

Andy
User avatar
Administrator
Posts: 12629
Joined: Fri Aug 18, 2006 11:47 am
Location: United Kingdom
Contact:

Re: The server is held together with cron jobs!

Post by Andy »

I can imagine it being a bit like this... lol.

Image

Nah, with all seriousness, these crons are necessary to make my life easier, otherwise I'd be sat there running scripts and restarting software all the time to keep it all 100% fully working.

Tilka
User avatar
FTP Access
Posts: 127
Joined: Fri Oct 17, 2008 9:51 pm
Location: Germany

Re: The server is held together with cron jobs!

Post by Tilka »

I like your "that process crashes from time to time so let's just kill it from time to time" attitude.
This signature intentionally left blank

Andy
User avatar
Administrator
Posts: 12629
Joined: Fri Aug 18, 2006 11:47 am
Location: United Kingdom
Contact:

Re: The server is held together with cron jobs!

Post by Andy »

Has to be done! Its the easiest fix as well rather than solving the problem. The problems are in the software itself, which I didn't make, so I can't fix those problems, only the software creator can and in most cases they won't. In the case of Filezilla I think its the sheer volume of data passing through it that eventually causes the issues. It doesn't have a memory leak, it just stops accepting connections properly. mrpijey discovered this a few weeks ago and since restarting it, its been flawless again and speeds have increased.

Rob Jansen
User avatar
Donator
Posts: 5271
Joined: Sat May 12, 2007 1:05 pm
Location: The Collection Book
Contact:

Re: The server is held together with cron jobs!

Post by Rob Jansen »

happy dude wrote:This is why OVH don't release pictures of their datacentres. You'd be horrified if you saw the physical state the servers in
They do have an video of there datacenter.

Al is put in racks and neatly put away.

And If I am downloading with Filezilla, the only time I know about the cron job is that the download restarts at the moment it would disconnected (well resumes).

Then the timer get's reset, so I was downloading for 10 minutes and it was back at 0 minutes.

Andy
User avatar
Administrator
Posts: 12629
Joined: Fri Aug 18, 2006 11:47 am
Location: United Kingdom
Contact:

Re: The server is held together with cron jobs!

Post by Andy »

Yeah, the restart only happens once a week though, every Sunday at 4am.

Ambig
FTP Access
Posts: 428
Joined: Mon Jan 28, 2008 4:53 am

Re: The server is held together with cron jobs!

Post by Ambig »

What actually runs the cron jobs? Is there specific software out there that does it, or is it included in Windows Server? (I'm assuming you're using Windows Server, possibly 2003?).

Also, when you kill these services, what starts them back up? Do they just startup again automatically, or does the cron job take care of it?

Andy
User avatar
Administrator
Posts: 12629
Joined: Fri Aug 18, 2006 11:47 am
Location: United Kingdom
Contact:

Re: The server is held together with cron jobs!

Post by Andy »

Its task scheduler in Windows. You can get software to do it but when windows has a service for it why bother?

When a task is killed, its one that is opened by something else or another cron. PHP for example is executed by the web server as and when a page is needed, so that opens again by itself even if a cron kills it. Netgraph for example needs to be opened by another cron, which is why after its killed its started again by the same cron job.

Ambig
FTP Access
Posts: 428
Joined: Mon Jan 28, 2008 4:53 am

Re: The server is held together with cron jobs!

Post by Ambig »

Ah. I see.

I've never messed with Task Scheduler, so I didn't even know what it can and can't do.

Andy
User avatar
Administrator
Posts: 12629
Joined: Fri Aug 18, 2006 11:47 am
Location: United Kingdom
Contact:

Re: The server is held together with cron jobs!

Post by Andy »

Its simple really, you just tell it what file to run, at what times and how often. Nothing more to it than that except if you wanted to change permissions too, which you probably wouldn't in most cases. Its fairly simple to understand.

zamadatix
User avatar
Donator
Posts: 2381
Joined: Tue Aug 12, 2008 7:37 pm
Location: United States

Re: The server is held together with cron jobs!

Post by zamadatix »

Andy wrote:Its the easiest fix as well rather than solving the problem.
This is a good quote, especially when the background information is not there .

expert01
FTP Access
Posts: 352
Joined: Thu Aug 24, 2006 6:28 pm

Re: The server is held together with cron jobs!

Post by expert01 »

I'm trying to get the cash together to rebuild my server (which just lost its 1TB drive) with a cheap quad core and a RAID card. I'm planning on Server 2008 with the Windows Home Server components (I can't believe no one's really tried yet to rip the files from the Home Server installation disc and make a standalone copy that can run on regular server yet); it'll be my web server, file server, backup server, download server, FTP server, firewall, virus scan and proxy server, router, and whatnot.

Anyways, I'll see if I have any good tips to give you afterwards. My linux server ran like a rock, restarting only when I let it install updates (until recently when it needs restarted every time lighttpd can't access the 1TB that's gone). My desktop only restarts when I install an update or a program, or those rare times that a game crashes. My last windows server almost never got restarted. Processes really shouldn't be crashing like that.


What really ticks me off is that the largest web programs out there, apache/php/mysql, require tedious manual tuning to run at peak efficiency. My $10 computer games automatically pick the best graphics settings for me, why can't these programs benchmark my system and tune everything for me, maybe doing some dynamic adjustments to keep the server up? Why hasn't someone made a program that will automatically balance resources between those three programs for max efficiency? You'd think they'd make a killing with something like that.

Thlump
User avatar
Donator
Posts: 890
Joined: Wed Aug 27, 2008 12:52 am
Location: United States

Re: The server is held together with cron jobs!

Post by Thlump »

Very interesting! It's simple to coordinate but is very very effective. What I don't understand is why some applications don't free up memory that it no longer needs
Image

Andy
User avatar
Administrator
Posts: 12629
Joined: Fri Aug 18, 2006 11:47 am
Location: United Kingdom
Contact:

Re: The server is held together with cron jobs!

Post by Andy »

Thlump wrote:What I don't understand is why some applications don't free up memory that it no longer needs
Because the coders didn't design it that way, or its not possible with the environment they used as its not advanced enough.

Post Reply