Cortanagate

As well as SharePoint and Office 365, my company does quite a bit of mobile development, particularly on Windows Phone which I still think is the best phone OS. We have a number of Windows Phone apps in the Windows Phone Store, including a popular Dictionary app. The app is free because, frankly, charging each user $0.99 (which seems to be the going rate for apps nowadays) hardly seems worth the trouble either to us or to our users. We would rather have the reach and maybe get a little bit of publicity and a few free phones from Nokia Microsoft @Lumia.

Dictionary recently passed two million downloads. For a Windows Phone app, that's not too shabby.

We get quite a few messages telling us that this or that word is missing, or the definition isn't quite correct. We don't have a team of lexicographers of course - the definitions come from a number of sources including the WordNet database and GCIDE which is an XML version of CIDE which is itself derived from the 1912 version of Webster's dictionary. Both are American and we have added the non-US spellings (which are used in most of the rest of the English-speaking world and not just Britain). Inevitably we have missed a few of these, and from time to time we get an enraged message from a British user about our silly US spellings. It's too embarrassing to admit to them that we are based in England.

After a while you get used to the occasional vitriolic review in the store. You might think that since you are not charging people for the app that they would cut you a bit of slack, but no. Sometimes we get the full treatment. People get very passionate about their phones - it's their personal space and when they install your app they are putting their faith in you to some degree. If they feel you have let them down they can take it quite personally. If you give them an update and it changes something they liked, they don't have the option of going back to the previous version like you do on a PC.

So I rather dread updates, because the way the store works if anything goes wrong you are left at the mercy of the people who manage the certification process. On the whole they do a very good job and I think they are probably the best of the app stores in getting the right balance of supporting developers without making it a free-for-all. But things slow down from time to time. You can get trapped with a problem update and with no way of rolling it back. Meanwhile your users have no way of rolling back either, and so you are just stuck there waiting and apologising to people.

Last Thursday, after much testing, we submitted version 3.9.1.0 of Dictionary that includes some content updates and a significant improvement that adds Cortana voice commands so that users can say things like 'Dictionary define banana' and it will start to speak the definition back to you. We have about a dozen devices for testing with various versions of Windows Phone OS, as well as a few beta testers, and the updated version worked perfectly on all of them. After a few hours the app was certified and on Friday morning we published it. We monitored the reviews in the app store throughout the day just in case there was a problem we had missed (we typically get about 40 reviews daily). By late evening everything was looking okay.

On Saturday morning there were the usual five and four star ratings and reviews (thank you everybody) and the occasional bad rating. Occasionally we see a negative review or something bewildering like 'Fantastic app - highly recommended' along with a one-star rating. But there were also three rather worrying one-star ratings with comments along the lines of 'app closes immediately' and 'won't load following the update'. Eughhhh!

And so the nightmare begins. We have an increasing number of users with an app that now won't load, but we can't reproduce the problem. All we can do is ask them what device they have and what OS version and start looking for patterns. It appears to be a small percentage of users who are mainly using Nokia devices with the model variant RM-914 and RM-885. You have to do a bit of detective work to know that these are Nokia Lumia 520 and 720 devices which both shipped with Windows Phone 8.0 but can of course be upgraded to Windows Phone 8.1. We briefly upgraded our project build to 8.1 before manually editing the build configuration back to 8.0. Could the problem be that these users had phones that were still running Windows Phone 8.0? One of our test phones is still running 8.0 and that was still working, but maybe it was a specific 8.0 build.

Fortunately a few users emailed to let us know which version of the OS they were using. They were using Windows Phone 8.1 which ruled out the Windows Phone 8.0 theory. To further confuse matters, one user had a specific OS version and phone model that matched one of our test phones which was working perfectly. Yes they tried reinstalling the app. Yes, of course they tried restarting the phone.

Now we have a problem out in the field and no way of reproducing it. We were reduced to looking at the lengthy change log from the previous release and trying to guess what the cause might be. We took a guess that an API we had just started using, which retrieves a file containing the voice commands configuration, might be the cause. It hadn't given any problems in testing but maybe there was some combination of language packs that caused it to throw an exception. Maybe it was Cortana's fault. We decided to silently 'swallow' any exceptions thrown by this API call; not something developers are encouraged to do, but since we didn't know what was causing it we couldn't think of any useful action to take on behalf of the user. We just needed to get this thing working again as quickly as possible.

We created a beta version of the application to test the theory and a few users who we knew had the problem agreed to test it. To our enormous relief they emailed back to say that the beta version worked, so it looked as though we had a fix. Now all we had to do was publish the updated version to the store and all would be well.

Except it was Saturday.

A common developer request for the Windows Store is to be able to roll back a release to a previous version. We are told that this would be very difficult to implement, which I do not doubt, and it is also argued that since the certification process is only taking a few hours nowadays, then it is easy enough to simply re-publish an older version. That might be true in the middle of the US working day. But on Saturday it seems that nothing happens.

Nor on Sunday. We had managed to get a fix within hours of being alerted of the problem, but then had to wait until late on Monday (UK time) for an update. In the meantime the problem version of the software sat in the store being automatically loaded onto our users phones and stopping the app from working. In the meantime we had to watch the one-star reviews roll in as disappointed users wondered why it wasn't working any more, and why it was taking us so long to fix it. We even had emails: 'I can't live without this app'; 'Please fix ASAP - need this'. It was an obscure problem, but it could have been a quick fix - instead it became our very own Cortanagate.

I hope this post is of interest to our users who probably, at times like this, ask themselves what is going on behind the scenes (if anything). We do try our best. It may be a free app but it's got our names on it and it's heart-breaking to see the one-star reviews mount up when there is nothing we can do about it. And we do appreciate the understanding of our users and particularly those who take the time to try beta versions and email us with information that helps us solve the problem. I hope the Windows Phone Store people read this too, and perhaps get some idea of the frustration and difficulties that arise when an update goes wrong. I understand that they need to be able to scale, but if developers are to keep updating apps then it needs to be possible to rectify mistakes, so that releasing updates doesn't call for the conservative engineering practices of a manned space mission.

If you would like to register for the next beta version please email info 'at' flosim.com and give us the email address of the Microsoft account you use on your phone. There is an upper limit on the amount of beta testers we can have, but after this experience I think it's a case of "the more the merrier".