The Mysterious Case of Comment Breakage
by
Can you tell I've been reading Mark Russinovich too much lately? Yeah, I'm getting kind of annoyed by it too, but anyway..The comments have been broken for almost two months, and only last night did I find out this lovely fact. They are fixed site-wide and you are free to spew forth and enlighten me on the many errors of my ways. If you're wondering what the problem was, and feel really, really bored today, then read on, as I shall explain in truly excruciating detail every thought I had and thing I've done since the day I was born that lead up to this terrible calamity. Last night while I was culling news for a big posting session the likes of which I haven't done in months -- we're talking 6-10 posts instead of one or two -- I got an email from a reader letting me know that my post on Jericho coming to the Scifi Channel had bad info in it, that my comments were broken, and that my contact page was devoid of any actual contact information.
Wow, kind of makes you want to crawl into bed early and pretend somebody else was sitting at your workstation all day, you know?
Fixing the page with contact information was the easiest, obviously, as I just copied the information from another page hosted with Google that I've been referencing in my E-Mail signature for months. That single page was the only "filler" page on the entire site that never got any actual content on it back when I launched the blog in July, and explains why nobody has bothered to tell me that I broke the comments since the beginning of December.
I know it has been exactly that long, but I'll get to that in a minute.
Veteran aircraft accident investigators often say that disasters never happen for a single reason, but that they are always a confluence of many events that come together at exactly the same time to screw up your day.
This is true.
First, I abandoned any hope if posting the news I had just culled after getting the E-Mail and spent the next 1.5 hours tracking down the cause. The problem I knew almost instantly: any attempt to comment resulted in a standard error message indicating that you didn't pass the CAPTCHA test. You're familiar with these -- it's an image with random letters and words that make you prove you're a human being, and not a spam robot.
After looking at the HTML source for the most recent post, I noticed immediately that the CAPTCHA test wasn't in the page you see when you read a blog post, but the script that checks everything in the back-end -- the part that actually does the test -- was still working just fine. Since nobody was presented an opportunity to enter the correct letters and numbers, everyone failed the test and their comments were rejected.
Good news: I've seen how lame the error message is and I intend to replace it with something much more user friendly sometime this weekend, so people will better understand why that failure happens, and what they can do about it.
After a full 1.5 hours of examining my site templates, the actual PERL code for the CAPTCHA test, checking the release notes for my last version upgrade, the notes for the newest upgrade, and finally upgrading, I found the problem.
I'd note that upgrading didn't fix it, but it did fix some missing icons on my blog's dashboard. Cool!
Indeed, for some reason that escapes me even today, the template holding the comment stuff had a stray tag in there that was preventing the CAPTCHA stuff from being inserted into the page. I had not edited that comment since the blog launched, and yet I've had over 100 comments previously so I know this code used to work at some point.
...
<MTIfCommentsAccepted>
<MTIfRegistrationAllowed>
...
<MTElse>
<$MTCaptchaFields$>
</MTIfRegistrationAllowed>
</MTIfCommentsAccepted>
...
What this code boils down to is the following logic:
If Comments Are Allowed on This Post, then...
If User Registration is Turned On, then...
[apparently do nothing because this area is empty]
else do this..
[insert captcha code]I still may have to mess with this code because even now, it still doesn't look right, but I did strip out the "else" statement so all you've got are two true-or-false tests. I suspect that "else" actually belongs in there along with some other missing stuff, and that was the real problem. Regardless, the change I made restored commenting functionality for the time being, and that's why the test wasn't appearing under the comment form, and subsequently one of the two reasons why the test was failing and comments were not being allowed.
I dug a little deeper than that to see if I could pinpoint exactly when comments became broken, to see if it would jog my memory. I really wanted to know if I had done this, or if this was a result of a botched upgrade, or some other devious activity.
The last comment had been posted in the first week of December, the 5th I think it was, and none had been posted since. Now look, this blog is small and comments were rare to begin with. In the early months it was getting a 1-4 comment to article ratio, or even with nearly two months of no comments at all, about ~100 comments for ~550 posts. When I stopped seeing comments with already low numbers, I simply assumed it was because people had lost interest in commenting because I was writing about some lame shit nobody cared about.
It never occurred to me that comments were broken, since I had a "contact" page with all my contact information on it, right?
Not so much.
That got fixed and now you can gather my E-Mail address and no fewer than five instant messaging identifiers off that page.
So obviously sometime in the first week of December, I apparently broke my template which broke comments. Only that's not what happened at all. I still don't know how the template error got in there, or when, but while trying to figure that out, I found another error on my part that also broke the CAPTCHA tests that I also wouldn't have otherwise known about.
This explanation requires some back story for those who don't know how a blog like Movable Type works behind the scenes, and understanding how it works will explain why I did something the way I did.
Movable Type came first, and worked by storing information in a relational database such as MySQL. When you "published" a post, all the information such as post title, the actual post text, author information, the sidebar, all that stuff was culled from the database and written into a very average "static" HTML file on the server hard drive. Whenever you read a post on a Movable Type blog, you're loading a plain-jane webpage which the webserver reads from the hard drive, and spits at you over the wire. This is simple, and it is very fast. Whenever you update that post, or whenever someone adds a comment requiring the file to be updated, Movable Type simply re-writes the file to disk.
WordPress came along during the PHP boom when people really didn't use PHP for high-traffic websites. At least, not like this. Most dynamic sites using PHP that do cull information from a database don't make many calls, and don't need very much actual data. Additionally, most of them don't get hundreds of thousands of hits per day.
So here we have a new weblog entrant that is entirely dynamic. You request a page like this: http://www.example.com/?p=56 -- where you're telling the webserver to give you page 56, and the PHP go to work actually building that page on-demand, just for you. It hits the database pulling out the post information, the author information, the comments, the author information for each commentator, etc. And it does this for every single page request, using anywhere from 7-30 database requests to build a single page, for a single visitor.
That's fine for a few hundred hits per day, but for 300,000 per day, it requires exceptionally fast server hardware to handle that much active load. It is constantly busy building pages with totally redundant information.
Call me biased, but I chose Movable Type over WordPress because the publishing model is vastly superior. When your database goes down with a WordPress install, your entire website disappears. With no access to post data, it can't build a page for each visitor.
Since like WordPress, the Movable Type backend is entirely driven by dynamic scripts that access a database, it too is disable in such a situation, but only the back-end. All the HTML files (one per post, one per category, one archive per month, etc) are still sitting happily on the server's hard drive, ready to be served at a moments notice, with no processing required to be done on them.
So now you're wondering how any of that will help you understand the second part of CAPTCHA tests being broken. Here it is, the only real downside to Movable Type: if you want to change how your site looks, you have to change the template (stored in the database) and then republish every single post, so the existing files on the hard drive are overwritten.
That's the rub. WordPress spends half a second building a webpage for every single visitor, while Movable Type spends that time building the webpage once, so it can save it on the hard drive. If you change a WordPress template, every subsequent visitor will see the changes immediately for every page on the site, simply because there are no actual pages on a WordPress blog. Any post published through Movable Type will remain the same on the hard drive as the day it was published, until you write a new copy, which you must do if you change something like the "theme", or something that might appear on every page for the site like the sidebar.
Here-in lay the rub: because of the way Movable Type works, my blog can sustain far more traffic on a shared webserver than a WordPress blog can, and folks, it ain't even close. I can scale from 1 hit per day up to over 50,000 hits per day, and have no increased load due to Movable Type, because Movable Type isn't doing anything for all those visitors.
MT, in a basic since, creates the content. The webserver itself serves it to the public. WP creates the content for every visitor, so if WP takes half a second to make a single page for a single person, that's half a second the server can't spend serving other people. Serving a simple static webpage, believe me, takes a fraction of that time. It's not even comparable.
But, being stuck inside a shared hosting environment has its costs. In order to keep any one user from consuming all the resources a server can provide, the host has to watch what people do and if necessary, stop something from running (such as scripts) that are abusing the limited resources available.
I've recently run into this problem where the process of republishing my entire site was being halted before it had finished, as it was using as much free CPU as it could for minutes at a time. Though I've recently found a way around this, back in December, I thought I'd try something different.
I installed Movable Type on my home workstation, installed and synced its database with the one on the server, and used my own computer to publish the HTML to my own hard drive. I compressed the HTML when it was done, uploaded it to the server, and uncompressed it there.
Simple enough. It let me use the full resources of my workstation to build all this HTML without having to compete with other peoples scrips, and allowed me to do this an unlimited number of times, whenever I wanted, greatly increasingly my ability to make incremental changes to the site templates (how they look and function.)
Unfortunately, as I learned last night, the CAPTCHA information gets written into these static HTML files in such a way that the images must also exist in a certain place, in a certain way, at a certain time.
In other words, my local installation of Movable Type was generating the code for the CAPTCHA tests okay, but the images for those tests (and how that code was tied to the image via the database) wasn't getting moved.
No matter what I did with the templates (the first fix) it wasn't going to work that way, and thankfully I found that out at the same time as the template problem.
As I said, I've since found a way to rebuild the site incrementally (essentially I tell it to rebuild 100 posts at a time instead of all of them at once) so the remote building isn't an issue anymore.
Hopefully, all is well and fixed and I've already seen a couple of comments from last night and today. This weekend, I'll rebuild the site again (this will never effect your ability to view and read the posts) with some more helpful information on the comments area.
For example, there's supposed to be a link down there that allows you to register with the site, or with OpenID/TypeKey (nifty distributed login stuff) and if you login, you don't have to deal with the CAPTCHA test at all. Not only is that information missing, the link is missing too.
Not cool, but at least comments working again...again. (Yeah, this is the second time I broke them. So, did I tell you I applied for a job at the federal reserve recently?)
January 24, 2008 12:21 PM | Reply