tl;dr

The moral of the story is, the internet likes it when I moan.

Personal

Man there is some great stuff in here. Dire Straits! Although sadly Sultans of Swing — I’d prefer Romeo and Juliet, Brothers In Arms, or even (oh hell yeah) the ten minute live version of Telegraph Road. That’s one my dream rhythm game bands; now lets see some Pink Floyd (Comfortably Numb!) and Sugar or Husker Du.

Wait a minute, Comfortably Numb is in that list! (crosses fingers) please be real, please be real, please be real….

Now when am I going to be able to buy it here in the UK, eh?

And now – courtesy of CVG via PS3Fanboy, the entire list, with artists filled in.

Warmup

• If You Wanna get to Heaven, Ozark Mountain Daredevils
• Gold Lion, Yeah Yeah Yeahs
• Passum Kingdom, Rubberneck
• Black Betty, Ram Jam
• Don’t Speak, No Doubt
• Black, Pearl Jam
• Drain You, Nirvana
• Farmhouse, Phish
• Santeria, Sublime

Apprentice

• You Oughta Know, Alanis Morissette
• We Got the Beat, Go-Go’s
• Float On, Modest Mouse
• Kiss Me Deadly, Lita Ford
• Atomic, Party Animals
• Heartbreaker, (Could be so many artists)
• Piece of My Heart, Big Brother and the
• Holding Company
• Rock the Casbah, The Clash
• I Wanna be Sedated, The Ramones

Solid

• Panic Attack, The Paddingtons
• Chop Suey, System of the Down
• Give it Away, Red Hot Chili Peppers
• Everlong, Foo Fights
• Kids in America, Kim Wilde
• Ace of Spades, Motorhead
• Hello There, John Prine
• Anyway You Want It, Journey
• Pinball Wizard, The Who

Moderate

• Machinehead, Bush
• Born to Run, Bruce Springstein
• What’s Your Frequency Kenneth, REM
• Want To, (not a clue)
• I Write Sins Not Tragedies, Panic at the Disco
• The Old Apartment, Barenaked Ladies
• When I Come Around, Greenday
• Zombie, Cranberries
• Soak up the Sun, Sheryl Crow (?!)

Skilled

• Waitin on the World to Change, John Mayer
• Man in the Box, Alice in Chains
• Ramblin’ Man, Hank Williams
• One Step Closer, Linkin Park
• Misery Business, Paramore
• American Woman, The Who
• White Wedding, Billie Idol
• (I Can’t Get No) Satisfaction, Rolling Stones
• Folsom Prison Blues, Johnny Cash

Challenging

• Our Truth, Laguna Coil
• Are You Gonna Go My Way, Lenny Kravitz
• Bring Me to Life, Evanescence
• Spoonman, Soundgarden
• Head Like a Hole, Nine Inch Nails
• Aqualung, Jethro Tull
• Double Vision, Foreigner
• Rodeo, (your guess is as good as ours)
• Sultans of Swing, Dire Straits

Blistering

• Magic Man, Heart
• Bad Medicine, Bon Jovi
• Every Little Thing She Does is Magic, The Police
• Thanks for the Memories, Fall Out Boy
• Testify, Rage Against the Machine
• Kickstart My Heart, Motley Crue
• I Was Made For Lovin’ You, Kiss
• Bad Reputation, Thin Lizzy
• Prodigal Son, The Rolling Stones

Nightmare

• The Bleeding, Five Finger Death
• Down With the Sickness, Disturbed
• Hyper Music, Muse
• Judith, A Perfect Circle
• TNT, AC/DC
• Rainbow in the Dark, Dio
• Anna Molly, Incubus
• November Rain, Guns ‘n’ Roses
• Comfortably Numb, Pink Floyd

Impossible

• Feed My Frankenstein, Alice Cooper
• Peace sells… but, Megadeath
• Fuel, Metallica
• Crazy Train, Ozzy Osbourne
• Forsaken, Dream Theater
• 2 Minutes to Midnight, Iron Maiden
• Hot For Teacher, Van Halen
• Cowboys from Hell, Pantera’
• (Ghost) Riders in the Sky, (any number of people)

Personal Games

I received a reply from O2 to my recent complaint:

Hello Richard,

Thanks for your email about your upgrade order.

We didn’t receive your upgrade order. Your order wasn’t accepted, this is the reason you didn’t receive an email confirming the status of your order.

I’m sorry to hear you’re disappointed with the level of service you’ve received. We value our customers and we ve (sic) tried our best to provide you with the best possible service.

We’re working as fast as we can to deal with the high volumes of upgrade requests we received, but we cannot confirm for you at the moment whether your upgrade was successful. We recognise that this has not been a brilliant experience and apologise for the obvious frustration, but we are doing everything we can to resolve the situation as soon as possible.

Demand for iPhone 3G is staggering. We invested heavily in our website capacity which was tested carefully in advance, but we were experiencing 13,000 orders per second being placed, far beyond our expectations and our  worst case  scenario.

This may be of little comfort to you, but we were as prepared as we could possibly be but the sheer volume of demand is completely unprecedented.

We made a limited allocation of iPhone 3G stock available for pre-order online, primarily for those customers that pre-registered their interest. Demand has been very high and we have now sold out of this allocation.

To upgrade to the new iPhone 3G, please visit your nearest O2 store. Please be reassured that for new and eligible to upgrade O2 customers including iPhone existing customers, there will be iPhones available in store from 8:02am on the 11 July, although we again expect demand to be very high, so urge you to get down there early.  All iPhone stock is being sold on a first come, first served basis.

You can also upgrade to the iPhone from Carphone Warehouse stores from 8:02am on 11 July 2008. Please note that you won’t be able to upgrade from an Apple store.

Now to be fair this is suitably apologetic but I take umbrage at two points.

Firstly, what this letter boils down to is “we messed up our website and that wasted a day of your life, so here’s an idea: go queue up outside one of our stores instead. Oh, they won’t have many though, so you’d better get there at ungodly o’clock. We’re really sorry. Please buy one.” Is it just me or is that quite insulting?

Secondly, that 13,000 transactions per second figure. Now, my posturing in my complaint letter wasn’t unfounded; I really have done scalability testing and analysis for some of the biggest travel ecommerce solutions in the UK. I will happily admit that 13,000 per second is a hell of a lot of traffic. Wow! 13,000 per second! I cannot imagine enough servers to cope with that; well, that gets O2 off the hook then. Quite understandable.

But wait a goddamned stinking minute. This doesn’t add up. In his letter to various customers, O2’s CEO Matthew Key said that

To put it in context we had over 200,000 people expressing interest and only a very small proportion of that number of devices available. Faced with this dilemma, we made it clear in the communications that to be fair to all customers the orders would be managed on a first come first served basis, as stock was limited. The response was so great that the online store completely sold out of iPhones within just a few hours.

Now, I’m nowt special, but I’m pretty sure 200,000/13,000 = 15.6. In other words, if the O2 website was processing 13,000 orders per second at its peak we would expect all 200,000 customers who asked for O2 to contact them about pre-orders to have ordered in a single 15 second period. Let’s be generous though; it was 13k peak and not 13k sustained, and it was “over 200k”. That still clearly implies that every single one of those pre-registered customers would have had to gone onto the site within something around a two minute window though. Furthermore, as there were only a “very small proportion” of those that could order before stock ran out, the stock should have been exhausted in, say, less than a couple of seconds. O2 have confirmed verbally to me and in emails to a few bloggers that stock lasted through until 11AM or so. So that doesn’t make any sense.

It also would require O2 to have simultaneously delivered the “hey, come buy me!” teasing SMS to all 200,000 people and for all of those people to be sat right at a computer and immediately gone “woo, here I come!”. In fact, I can confirm anecdotally from a small sample of friends and bloggers that those SMSs were received anything between half six and about half eight Monday morning. This 13,000 per second figure has been widely cited, by places such as Reuters and Daring Fireball, but seems to me to be scarcely credible.

Y’know what I think? I think its a cascaded failure of the system, a failure mode I violently feared when I was scalability analysing. Basically, it goes like this. In the first ten minutes a couple of hundred people try to order. In the second ten minutes a few more hundred new people arrive, plus half the first group whose first attempted order failed. Thirty minutes in and you have a thousand people banging on the system and now it’s really in trouble. Draw that curve out and you end up at 13,000 hits per second — entirely consisting of people on their tenth, twentieth, thirtieth attempt at ordering.

See, that’s a much less sexy headline. Suddenly we’re not “O2 took 13,000 orders per second and the servers melted to slag, but hey, no mortal could cope with that and Mighty Thor was on his day off”. Now we are “O2 took a hundred orders in the first five minutes and the system crapped out for most of them, then those people tried again and more people came and oh my it’s dying”. Now perhaps that doesn’t sound any different but the point is that this second scenario, which I contend is much more likely to be what happened, was both predictable and preventable. O2 knew how many SMSs and emails they were sending. They could easily figure the maximum possible rate customers could arrive at the site, and should have specced hardware to cope with this load, plus a bit of headroom. They didn’t do this, because if they did, there is no way on God’s green Earth they would have reached a load of 13,000 transactions per second.

I would dearly love for O2 to open its kimono and show some log analysis from the Apache servers. I’m willing to give them a fair crack of the whip, I’ve been around the block with this stuff and I know how subtle and tricky it can be. I’d subject it to rigorous analysis, not some made up superficial blogging crap. Bet they won’t do that though. Time and time again in my last job I dealt with clients with no idea about scalability analysis, who wanted to stick a finger in the air and leave a one hour meeting with all the figures somehow conjured up. They were always disappointed when I took weeks of running simulations and load tests before I would commit to any numbers — but none of the sites I looked after ever went down like this one did. I’m willing to bet O2 tried to wing it, jimmied some random number of servers into the cluster, and are now trying to wriggle out of the PR with some old-fashioned “look how big the numbers are!” pseudoscience and “hey, Apple didn’t give us enough!” blame-oh-rama.

Well, I call shenaigans on this. I’ll go further in fact. I bet sometime over the last few weeks, somewhere in O2, some bigwigs met some techies and some Bob Techie said “Here is the little microsite we made to take iPhone preorders.” Jim Sarky Technical Architect said “You fool! If that site crashes, what will happen to www.o2.co.uk?” That made Fred Bigwig nervous and he said, “Shit dude, yeah. Put the microsite on its own server cluster.” But of course there wasn’t a server cluster around for this, and Fred Bigwig wouldn’t sign one off, because who buys a cluster for just one day of taking orders? So, Bob Techie did what techies do, and he improvised with some spare servers, and maybe reallocated them from QA, and reused some old ones, and generally lashed a cluster together. And minutes after those SMSs went out the bailing twine and spit that was holding it together fell apart and now here we are, either writing (me) or reading (you) a whiny blog post about it made out of overlong sentences.

Oh, and finally, none of these shenanigans around scalability explain why the system was taking orders after stock ran out or why the frontline support staff had no visibility into the order system. I still think both of these things stink. From where I’m sitting it still looks like a world class balls up.

Personal iPhone, Rants, Tech

I despair at Windows Mobile, I really do. Here’s just the latest in a two year series of issues, hiccups, problems, and bloopers.

I was planning on buying an iPhone on Friday (I say was…) and so I turned my attention to getting my contact and calendar data off my current HTC TyTN (running Windows Mobile 6.0) and into Outlook. From there, I can sync them through iTunes directly onto the new iPhone, saving me retyping all those hundreds of phone numbers, addresses, etc. Good times.

Now, this phone has been around a bit so it already has two sync partnerships on it; one with my workstation in my last job and one on an older personal laptop I don’t use any more. I want to sync it now with my new work laptop, which is also my primary personal machine. Naively, I plug it in, and to be fair it quite happily connects, runs off to Windows Update, refreshes the HTC_USB driver, installs Windows Sync Centre Doodad, and offers to create a partnership. So far so good — this is somewhat slicker on this Vista machine than it was on the previous XP machines. And then It Happened.

It was a really innocuous dialog that I really wish I’d taken a screenshot of that looked not entirely like this:

Sadly, I didn’t screenshot the actual screen I saw. It was quite similar to this one but instead of showing me the partnership from my laptop to the TyTN — the one I made earlier — it was showing me the two partnerships my TyTN was already engaged in when I started today — one with my old workstation and one with my old laptop. Clearly my phone is a sync slut.

Now, it turns out Activesync only supports two partnerships in total so the dialog I saw earlier didn’t quite look like that one above. It said “I only support two partnerships at once. Please delete one of these two” and then it listed “timora” and “Windows PC 2″ (I love that second name). Knowing full well I don’t have access to either of these machines again, I just clicked on one, then the other, selecting a nicely drawn Vista-ised bubble button down at the bottom that said simply “Remove Partnership”.

One of these — I forget which — took a long time to respond. That should have set alarm bells off but I was multitasking too much and didn’t see what was going on. Perhaps you, dear reader, are more alert than I and have guessed where this story is leading.

Anyway, on with it. I created my new partnership, synced across, and opened Outlook to find… no contacts. Eh? Closed and opened it again, no contacts. Disconnected and reconnected advice, watched Sync Centre go from “Disconnected” to “Syncing” to “Done”, no contacts. Rebooted laptop, reconnected device again, confirmed the sync partnership definitely covered contacts, opened Outlook, no contacts.

Opened Pocket Outlook on the device. No contacts.

Gngh.

It seems that when a Windows Mobile device acquires a contact record over a sync, the record is tagged where it came from. And when you delete the sync partnership? Boom, record gone. With no warning that this seemingly disconnected event at the other end of the GUI will be deleting the most valuable data your device has.

Stupid old me only had a six-month-old backup too (after all, Activesyncing is supposed to be your backup isn’t it…). On the good news side, my contacts data is pretty static so I won’t have lost much. On the bad news side, I’m sure I’ve lost at least one or two new phone numbers or birthdays that’ll cause me annoyance down the road.

More seriously, the backup was taken after the last time Activesync freaked out and duplicated all my contacts but before I went through and deleted the three hundred of so duplicate entries. So I have to do that again. I suspect some sort of VBA into Outlook is the answer here. (Aside: I had Palm PDAs for three years, and moved around between three models, and it never ever mismanaged a sync).

My central point remains, however, that a user interface capable of deleting your most central and valuable data whilst doing something that, on the face of it, is entirely unrelated to that data and doesn’t see fit to even warn you is just plain dumb. Go away please Windows Mobile.

Personal Rants, Tech

Dear Sir or Madam,

As your own advertising has been only too keen to point out, the iPhone 3G is one of the most significant new devices O2 has launched in years. After slow initial sales of the original iPhone, the price cut and sales boost have built huge pent-up demand for the 3G model. You were clearly aware of the scale of this demand, as you put in place a pre-registration process for potential customers to be alerted by SMS when preorders became available. Indeed, that SMS said “Demand will be very high, so it’s first come first served”.

It is therefore with astonishment and disappointment that I am forced to contact you to register my disgust at how incredibly badly O2, as an organization, has bungled your customer services during this highest of high profile launches.

I switched to O2’s SIM-only contract nearly six months ago with the intention of upgrading to the iPhone 3G when it was released. I registered my interest on the O2 website within hours of the product announcement. I was on the website and attempting to preorder my handset within an hour of receiving the SMS advertising that sales had begun. Unfortunately, as my purchase was an upgrade of an existing account, I was forced to use the upgrade website and not the one for new customers. I have read that the website for new customers remained operational, as did Carphone Warehouse’s website. Sadly the upgrade website did not fare so well.

From 9am to 4pm I made dozens of repeated attempts to fill in my details on the site. I was repeatedly presented with errors as your servers were clearly incapable of dealing with the load they were placed under. This was O2’s first failing: you clearly had an idea of user numbers, as you sent each potential user of the site an SMS, and yet you still failed to ensure there was enough capacity in the servers. In the past I have performed scalability analysis for such high profile ecommerce websites as ThomasCook.com; I know full well that designed scalable websites is not a black art. If a website I had specified had performed as badly on launch day as yours did, I would have resigned immediately.

Eventually, at around 3pm, I managed to make the website direct me to https://upgrades.o2.co.uk/failover/ok.html, a page which told me that my order had been received and that if there were any problems with my details I would be contacted. Shortly afterwards the website changed to a “no stock” notification. However, I had no confirmation number of any kind, no delivery date, and even after several hours I had not received an email from the site. Suspicious, I contacted your customer service department on 0870 600 3009.

After requesting my call be escalated to a manager, as first line support seemed to have no information at all, I was informed that you had actually sold out of all the iPhone 3G stock at around 11am so there was no way that I could have a valid order. It seems that the website continued to give out seemingly valid orders for many hours after stock had actually run out; it also seems you have only secured a laughably small stock allocation. This problem was very widespread; the manager I spoke to in your call centre said she had dealt with 41 phone calls in the last hour from people in the same position as I am. As far as I can see, your failover cluster did not have any sort of online stock level fulfillment against the main live server, and hence had no ability to know it was double selling stock. Meanwhile anyone who wasn’t an existing O2 customer was able to buy up that small amount of stock from the “new customers” website without any technical problems. Essentially I was at a disadvantage because of my customer loyalty; surely this is not good business?

The final customer service blunder came when I asked the manager in the call centre to try and confirm my order details, just in case my order had slipped through. She informed me that your call centre staff could not see details of the iPhone orders placed on your web site, and that the only way for me to check was to see if the money had been taken from my bank. How were these staff supposed to do their jobs and support online sales without any access to the information coming from the website? It frankly baffles me how poorly thought out this entire process seems to be.

The iPhone 3G launch should have been a high point in O2’s financial year, and it was certainly a device I was greatly looking forward to owning. Instead I have wasted hours of my life today battling your poorly designed systems in a futile attempt to persuade you to take sell me an expensive 3G device — on top of a report in this month’s PC Pro magazine that, in addition to having the worst 3G coverage of any UK network, O2 also has the worst 3G network speeds.

Sadly, I now feel like a complete idiot. I would invite O2 to please give me some explanations for what went wrong today and a reason why I should not simply request my PAC at once and migrate to T-Mobile, whose mobile data service and customer services departments are more than an afterthought.

Yours faithfully,

Richard Gaywood

Edit update 2008-07-08: I have now had a reply from O2.

Personal iPhone, Rants, Tech

Gngh. Release date is still “to be confirmed”. Which presumably has to translate into “sometime in 2009″. Presumably they are staggering the release dates because they can’t make the instruments fast enough for a worldwide launch, but that’s exceptionally annoying as I only want the disc. Can’t import it either on the 360.

Also interesting that, as for the European launch, the US launch of Rock Band 2 will be a timed exclusive for the Xbox 360. That’s also likely because they can’t make the instruments fast enough and I suppose 360 is the dominant platform — although there’s scope for it to sell very well on the Wii.

If Guitar Hero World Tour ships in Europe in the Sept/Oct timeframe I could see it picking up purchases from people frustrated waiting for either Rock Band 1 (still not out on PS3/Wii) or Rock Band 2.

Personal Games