Page 1 of 1

The server is held together with cron jobs!

Posted: Fri Apr 03, 2009 2:39 pm
by Andy
Well I said I'd get round to giving you guys the lowdown on this one, so here it is!

As you know the server does a fair amount of work on its own. It went 2 weeks without me even touching it while I was in hospital, something not many servers can do. The reason behind this is the cron jobs running behind the scenes keeping things running and fixing potential problems before they occur.

I have a number of cron jobs running at various times.

1 minute cron job
10 minute cron job
1 hour cron job
4 hour cron job
1 week cron job
So, what runs on each one? Most are self explanatory but I've put some comments in green for those that aren't so obvious.
1 minute cron job
The FTP statistics page update script
The mini-status update script
The memory usage update script
The CPU usage update script
The UPTIME update script
MRTG Traffic Graphs update script
10 minute cron job
Setting the CPU usage priority of the following processes to LOW:
PRTG Traffic Grapher.exe
DFRGNTFS.exe //Defrag, which doesn't get run often
WinSCP.exe
IRCXPro.exe
FileZilla.exe
uTorrent.exe
FileZilla Server.exe
php.exe // PHP4
php-cgi.exe //PHP5
sven1.exe // Game server process
sven2.exe // Game server process
gmod1.exe // Game server process
css1.exe // Game server process
Delete all files in d:\*.tmp // temporary files build up due to a php bug, so this deletes them
Delete all files in d:\public_html\betaarchive\*.tmp // temporary files build up due to a php bug, so this deletes them
Run the awshost status update script
Run the sessionfix script for the forum // without this the forum kept getting session errors, so this runs every 10 minutes regardless of any errors
Kill all cmd.exe processes running
1 hour cron job
Kill all php.exe processes // php sometimes crashes, so this kills all the processes once an hour in case this happens, so the max downtime is 1 hour
4 hour cron job
Kill all DeltaC.exe processes // this is the backups, which have a habit of crashing, so they get killed every 4 hours and reopened by the next cron job
Kill all rsync.exe processes // this is the backups, which have a habit of crashing, so they get killed every 4 hours and reopened by the next cron job
1 week cron job
STOP and START FileZilla Server FTP Server service // Filezilla drops connections when open a long time so this restarts the service once a week
Kill all netgraph.exe processes // when left open netgraph uses a lot of memory so once a week its killed
START netgraph.exe // this re-opens netgraph after being killed above
Most of the kill and start crons are designed to simply kill all unused processes that are left open by some operations, taking up memory. In the case of netgraph, it is killed and reopened once a week to clear old data from memory, as it doesn't do this itself. Over the course of a few weeks it takes up a few hundred MB of RAM. FileZilla Server seems to have issues keeping connections if open for weeks at a time, so a quick service restart of that, taking just a few seconds, solves this issue. It does mean anyone who is connected gets disconnected but its once a week so it doesn't really affect anyone that much.

So there you have it, thats how the server seems to fix itself!

Re: The server is held together with cron jobs!

Posted: Fri Apr 03, 2009 2:51 pm
by happy dude
For those who don't like reading, BetaArchive would simply like to advise its users that the server in its current state is held together with a combination of hot glue, masking tape and carefully placed rubber bands, so carefully placed that if even one were to be pulled or removed the whole server would disintegrate

and all that [censored]

Re: The server is held together with cron jobs!

Posted: Fri Apr 03, 2009 2:52 pm
by Andy
I've never heard it put like that before, but I suppose thats about right!!!

Re: The server is held together with cron jobs!

Posted: Fri Apr 03, 2009 2:53 pm
by happy dude
This is why OVH don't release pictures of their datacentres. You'd be horrified if you saw the physical state the servers in

Re: The server is held together with cron jobs!

Posted: Fri Apr 03, 2009 2:56 pm
by Andy
I can imagine it being a bit like this... lol.

Image

Nah, with all seriousness, these crons are necessary to make my life easier, otherwise I'd be sat there running scripts and restarting software all the time to keep it all 100% fully working.

Re: The server is held together with cron jobs!

Posted: Fri Apr 03, 2009 2:59 pm
by Tilka
I like your "that process crashes from time to time so let's just kill it from time to time" attitude.

Re: The server is held together with cron jobs!

Posted: Fri Apr 03, 2009 3:06 pm
by Andy
Has to be done! Its the easiest fix as well rather than solving the problem. The problems are in the software itself, which I didn't make, so I can't fix those problems, only the software creator can and in most cases they won't. In the case of Filezilla I think its the sheer volume of data passing through it that eventually causes the issues. It doesn't have a memory leak, it just stops accepting connections properly. mrpijey discovered this a few weeks ago and since restarting it, its been flawless again and speeds have increased.

Re: The server is held together with cron jobs!

Posted: Fri Apr 03, 2009 3:11 pm
by Rob Jansen
happy dude wrote:This is why OVH don't release pictures of their datacentres. You'd be horrified if you saw the physical state the servers in
They do have an video of there datacenter.

Al is put in racks and neatly put away.

And If I am downloading with Filezilla, the only time I know about the cron job is that the download restarts at the moment it would disconnected (well resumes).

Then the timer get's reset, so I was downloading for 10 minutes and it was back at 0 minutes.

Re: The server is held together with cron jobs!

Posted: Fri Apr 03, 2009 3:18 pm
by Andy
Yeah, the restart only happens once a week though, every Sunday at 4am.

Re: The server is held together with cron jobs!

Posted: Fri Apr 03, 2009 3:46 pm
by Ambig
What actually runs the cron jobs? Is there specific software out there that does it, or is it included in Windows Server? (I'm assuming you're using Windows Server, possibly 2003?).

Also, when you kill these services, what starts them back up? Do they just startup again automatically, or does the cron job take care of it?

Re: The server is held together with cron jobs!

Posted: Fri Apr 03, 2009 3:54 pm
by Andy
Its task scheduler in Windows. You can get software to do it but when windows has a service for it why bother?

When a task is killed, its one that is opened by something else or another cron. PHP for example is executed by the web server as and when a page is needed, so that opens again by itself even if a cron kills it. Netgraph for example needs to be opened by another cron, which is why after its killed its started again by the same cron job.

Re: The server is held together with cron jobs!

Posted: Fri Apr 03, 2009 8:19 pm
by Ambig
Ah. I see.

I've never messed with Task Scheduler, so I didn't even know what it can and can't do.

Re: The server is held together with cron jobs!

Posted: Fri Apr 03, 2009 8:20 pm
by Andy
Its simple really, you just tell it what file to run, at what times and how often. Nothing more to it than that except if you wanted to change permissions too, which you probably wouldn't in most cases. Its fairly simple to understand.

Re: The server is held together with cron jobs!

Posted: Fri Apr 03, 2009 11:34 pm
by zamadatix
Andy wrote:Its the easiest fix as well rather than solving the problem.
This is a good quote, especially when the background information is not there .

Re: The server is held together with cron jobs!

Posted: Sun Apr 05, 2009 7:54 am
by expert01
I'm trying to get the cash together to rebuild my server (which just lost its 1TB drive) with a cheap quad core and a RAID card. I'm planning on Server 2008 with the Windows Home Server components (I can't believe no one's really tried yet to rip the files from the Home Server installation disc and make a standalone copy that can run on regular server yet); it'll be my web server, file server, backup server, download server, FTP server, firewall, virus scan and proxy server, router, and whatnot.

Anyways, I'll see if I have any good tips to give you afterwards. My linux server ran like a rock, restarting only when I let it install updates (until recently when it needs restarted every time lighttpd can't access the 1TB that's gone). My desktop only restarts when I install an update or a program, or those rare times that a game crashes. My last windows server almost never got restarted. Processes really shouldn't be crashing like that.


What really ticks me off is that the largest web programs out there, apache/php/mysql, require tedious manual tuning to run at peak efficiency. My $10 computer games automatically pick the best graphics settings for me, why can't these programs benchmark my system and tune everything for me, maybe doing some dynamic adjustments to keep the server up? Why hasn't someone made a program that will automatically balance resources between those three programs for max efficiency? You'd think they'd make a killing with something like that.

Re: The server is held together with cron jobs!

Posted: Sun Apr 05, 2009 4:45 pm
by Thlump
Very interesting! It's simple to coordinate but is very very effective. What I don't understand is why some applications don't free up memory that it no longer needs

Re: The server is held together with cron jobs!

Posted: Sun Apr 05, 2009 4:46 pm
by Andy
Thlump wrote:What I don't understand is why some applications don't free up memory that it no longer needs
Because the coders didn't design it that way, or its not possible with the environment they used as its not advanced enough.