Warning serious NERD article ahoy!
Some major changes to the 'blog' fixing a few dozen problems, all very exciting stuff. Most confusing aspect was getting my lazily loaded images ( only the visible images load till you scroll the page ) working with read it later services like Pocket and Readability. I found a work around in the end for Pocket using PHP's $_SERVER['HTTP_USER_AGENT'] and detecting if it is PHP Pear grabbing my site, nasty I know, but the only way to do it for now. Pocket appears to use PHP behind the scenes to parse and so It's identity is returned as 'HTTP_Request2/2.2.0 (pear.php.net/package/http_request2) PHP/5.3.28' Readability is a bit nicer in that it reports as 'Readability/6534b2 - readability.com/about/' and Readability has the handy tag to ignore content 'class="entry-unrelated'
And I just added a comment system finally! I keep seeing people use Disqus so gave it a try. Turns out it cannot show more than one comment box per page which is no good, but found a work around... after a few hours
:-/
Nothing is ever simple.
Nothing is ever simple. In any case here is a zoomed out view of the code that runs the blog. The White code is the pre processor that is made of very squirrely looking AppleScript code and ran on my Mac. The grey code is a mix of PHP HTML and CSS which runs on the server side and reads in the XML produced by AppleScript. As you can see it is all a horribly ultra nested sprawling mess, though in my defense I did start this a decade ago and hadn't ever read a single thing on programming.
The whole blog is generated automagically out of folders with TextEdit documents in, just some RTF files that can hold Images and Videos. So I simple write an entry in a normal text editor, copy paste an image or video in. Done. And my AppleScript massages this into XML and export all the images, and videos, upload the videos to youtube and sync the whole site to the web host and so on.
Ah what kind of crazy bastard sets out and writes his own blog platform and website from scratch. Well Wordpress did'nt exist when I started, and I like images a lot on a blog, so copy pasting and ease of authorship was a must. Otherwise I wouldn't bother. But admittedly scary amounts of hours have probably been sunk into this over the years, bit by bit adding things.
I just made the whole applescript side multi threaded, take that crappy performing applescript code! It will now spin off a separate thread for each rtf file it is processing. Speeding the whole thing up by 100's of times. So that's nice. So no longer such a monolithic single block of code as I broke it up into several independent pieces. Go team me.
Below, See, makes perfect sense, The joy of escaping escape characters that are escaped. This abomination is what helps clean up an RTF file into something more legible, It somehow works
.jpg)
I also added caching for PHP page rendering using :
Link: www.phpfastcache.com --- www
As my site does a stupid amount of heavy string searching, replacing and other guff that is scary and abhorrent. So if say, I become really popular and everyone liked me so much as to visit my site 100 times a second, well the server would explode, and then my adoring fans would be deprived of access to my wisdom.
So typical processing of a page takes 0.2 seconds on a bad day. With caching... 0.001.
I wrote an AppleScript to spawn curl processes to assault my site as a test and total up the page rendering times with and without caching:
So for example with 5 users hitting the site every 0.1 seconds for a duration of 5 seconds you get:
39 seconds taken to run 5 seconds of access
5 concurrent accesses every 0.1 sec for 5 sec
556.525 seconds for sum of all load times for 250 page loads
2.2261 seconds avg per download
Without caching it takes 2.2261 seconds to serve up a page that took 0.2 seconds to render when only 1 render was being done at once, this will be due to contentions for disk access that are only revealed when multiple access happen simultaneously
So it basically took 40 seconds to serve up 550 seconds worth of page renders so it appears to be effectively achieving 13x at once ( on a 8 core Mac )
Now lets try with caching on:
7 seconds taken to run 5 seconds of access
5 concurrent accesses every 0.1 sec for 5 sec
17.886 seconds for sum of all load times for 250 page loads
0.071544 seconds avg per download
Nice, so now it took 7 seconds to run 5 seconds of access, bare in mind I am using the same machine to download the pages as serve them, it probably means it is coping 100% now
A difference of total page rendering time of 18 seconds versus the uncached 556 seconds of cpu time
It takes 0.07 seconds to serve up a page that took 0.001 seconds to render when only 1 render was being done at once now
Lets see how far I can push the cached load, given the over head of downloading from this many threads it likely the biggest issue now
20 seconds taken to run 60 seconds of access
5 concurrent accesses every 0.05 sec for 60 sec
8.178 seconds for sum of all load times for 6000.0 page loads
0.001363 seconds avg per download
CPU Load during serving up the cached pages on the left half, and uncached right half, plenty of room to spare for the cached, I suspect the red is the server and green will be me downloading the results. And my machine pretty much locks up while serving the uncached so.
Imagine how the poor typical 1 x 2ghz server would handle the uncached instead of a Mac Pro with 8 x 3ghz would not be pretty.
It would appear in any case that the majority of the cpu use is due to the overhead of downloading not the serving of the page. But it was a 'fun' exercise to stress test.
Now all of the above was tested on my local web server, lets try the remote Amazon EC2
With it uncached it hung the server really badly with some requests taking 40 seconds, and many just getting a blank return, only 300 out of 2000 came back! and those averaged 13 seconds per load
462 seconds taken to run 60 seconds of access
10 concurrent accesses every 0.3 sec for 60 sec
4818.08 seconds for sum of all load times for 373 out of 2000.0 page loads
12.917104557641 seconds avg per download
Lets run it again as I think my machine is struggling a bit with max proc limits
360 seconds taken to run 60 seconds of access
10 concurrent accesses every 0.5 sec for 60 sec
5007.7 seconds for sum of all load times for 370 out of 1200 page loads
13.534324324324 seconds avg per download
Interesting it craps out at the same 370 downloads, maybe some kind of spam protection on the server as I am loading all this from one IP
After thrashing the Amazon server, trying to access my site from a mobile phone ( so different IP ) fails completely. It is basically shutdown. Nice one amazon. And the whole time the Amazon status of the server shows as ok and not under much use.
Definitely odd as if I run again even a few small requests it takes 30 seconds per request ( I timeout curl after 30 seconds ):
88 seconds taken to run 1 seconds of access
10 concurrent accesses every 0.5 sec for 1 sec
560.322 seconds for sum of all load times for 18 out of 20 page loads
31.129 seconds avg per download
Below are a some cached runs thru, the first run thru is still showing hurt from the prior uncached stress test, after that they are fine again
11 seconds taken to run 1 seconds of access
10 concurrent accesses every 0.5 sec for 1 sec
31.288 seconds for sum of all load times for 20 out of 20 page loads
1.5644 seconds avg per download
8 seconds taken to run 1 seconds of access
10 concurrent accesses every 0.5 sec for 1 sec
0.288 seconds for sum of all load times for 20 out of 20 page loads
0.0144 seconds avg per download
9 seconds taken to run 1 seconds of access
10 concurrent accesses every 0.5 sec for 1 sec
0.389 seconds for sum of all load times for 20 out of 20 page loads
0.01945 seconds avg per download
9 seconds taken to run 1 seconds of access
10 concurrent accesses every 0.5 sec for 1 sec
0.336 seconds for sum of all load times for 20 out of 20 page loads
0.0168 seconds avg per download
And now for fun lets try a free web host ( opened host ) using the cached copy
11 seconds taken to run 1 seconds of access
10 concurrent accesses every 0.5 sec for 1 sec
0.104 seconds for sum of all load times for 20 out of 20 page loads
0.0052 seconds avg per download
Ok lets ramp it up for 60 seconds
152 seconds taken to run 60 seconds of access
10 concurrent accesses every 0.5 sec for 60 sec
3.476 seconds for sum of all load times for 1117 out of 1200 page loads
0.003111906893 seconds avg per download
ok so that's actually better than Amazon.... bastards
Lets try uncached now:
Webhost refuses to respond now
:-P
ok maybe not better than Amazon.. wait that does appear to be a DOS prevention as I can access site from my phone still but not the computer I was using to thrash the server
ok maybe not better than Amazon.. wait that does appear to be a DOS prevention as I can access site from my phone still but not the computer I was using to thrash the serverAnyway
I could just have used a service like load impact
:-P
which lets you test for free unto 250 users per second ( not connections but users with some kind of average page download per minute ) and pay for more based on use:Link: loadimpact.com
But it doesn't hit the server as hard as my own script does
When using LoadImpact at 250 users graph looks like this, resulting in 1478 page loads over 2 minutes ( When not caching page )
Ouch over 7 seconds load time, though my internet is running off crappy ADSL at the moment, waiting for fibre to be installed, and this doesn't include serving up the actual HTML and Images, that's just to process the page and return a number
And now with it cached:
Ok lets test my Amazon EC2 hosting
huh it is still loading stuff from my domain... *tries again*
Ah that is better
And with it uncached:
Identical, but I suspect the test was still cached
CPU graph on Amazon:
And for future references, here is what the site currently looks like:
