Finished the judging for the Esendex Developer Challenge today. A tricky exercise, the entries were diverse, in the end however we were all in agreement.
The winners are now safely tucked away into in a metaphorical gold envelope. As soon as I've spoken to them, we'll announce it to the world.
It's shiny, has a touch interface, makes a great cup of coffee and is now gracing the kitchen at Esendex Towers. Though everyone seems a little jumpier than usual.
Running an ecommerce site as we do, we get exposed to all sorts of fraudulent attempts to purchase our SMS services with stolen credit cards.
We've put all sorts of checks and restrictions in place to try and prevent this before we even submit the payment for processing by our merchant service but there's a background task to keep them up to date.
Sometimes I look at the number of people involved in development and strategy meetings on this subject and feel a mix of aggrievement and frustration.
Don't get me wrong it's not a big problem that we spend lot's of time on. It's just that we do spend time on it, it's just dead time. We're not moving the service on, we're just working to stand still. All because someone has decided that stealing is a legitimate approach to making a living.
Part of what drives me and the team is the gladitorial challenge of legitimate business competition. How can we differentiate our service, be first to provide the customer wants, be best at providing it. Fighting with low-lifes is no fun at all.
Rant over, thanks for listening
Now I've got the specifications of the SLAP protocol out of the way, I'd like to describe how we're using it and the benefits we are seeing. The first being the dynamic configuration of service end points.
In a service like ours we have a series of nodes that process messages and apply rules to manipulate and route them. If we want to change the location for a service, we have to first notify all the upstream services of the change of location.
In a planned scenario this can be laborious. All services that may make use of the service have to be reconfigured, often requiring a restart. It is possible to have these services monitor a config file for changes to avoid a restart but this doesn't remove the need to manually go through each service config and make the necessary changes. This becomes increasingly complex in a distributed system spanning many machines and networks.
The unplanned scenario is even less desirable. There are many reasons why a service can stop and this will generally be at the most inopportune times. Having to manually reconfigure services in response to a failure is not acceptable in a high availability scenario like ours.
The other problem with the unplanned scenario is that the client service has to deal with the unavailability at the point it's attempting to consume the service. Say for example this is a network service, it could be waiting for a network timeout before it recognises that the service is not accepting requests. It then has to fire-fight, cleaning up because it thinks the service is unavailable.
Far better for it to be told categorically that the service is unavailable, it can then queue requests or whatever has been coded as appropriate, while it waits to be informed that the service is operational again.
It is possible to use separate load-balancers to handle this kind of outage, but they cost money, need configuring, draw power and the solution the bring is by no means dynamic. I'll actually discuss load-balancing using SLAP in a subsequent post.
In the SLAP world, services are not configured to use fix endpoints like sip:ems3.prod1.esendex.com:8067 but rather are bound to well known service names eg: slap:smsrouter. The actual endpoints to the service are advertised in ANNOUNCE in order that clients can maintain a record of the current state of the services they need to consume.
If a client service wants to make use of an smsrouter service, it checks the state in it's local service state table before sending the request to the correct service URI. This information is kept up to date by the service. Most will announce their state on a periodic basis but I'd also consider it good practice for the service to send an ANNOUNCE when it's state changes, perhaps when it's under load or is shutting dowm, to ensure all clients are kept up to date.
The development team at Esendex have also found it very useful when building and debugging services as Jonathan describes in SLAP, my service’s up!. This was an unforeseen benefit but another one that has cut out a lot of the hassle with debugging services. The guys can step through the code, find the issue, write the unit test and rebuild very quickly which get's us to market far quicker.
More on load balancing soon.
In May this year, a group of us (parents, friends and teachers) from my sons' school are cycling from Nottingham to Paris in order to raise money for the school. My role in this exercise, in addition to cycling, is to build and manage a website (www.r4rh.org) to promote the event.
Part of that web site includes retrieving a set of latest photos from Flickr tagged r4rh or RideForRoundHill. I've built the site using ASP.NET so naturally downloaded the Flickr .Net API Kit. My needs were very simple, just searching for a list of photos, and it worked brillantly until I published the site.
For some reason, I don't know why, the reference to the FlickrNet.dll didn't see, to carry across. Initial investigations didn't uncover a solution, one of the problems I find with doing this kind of thing these days is that my integration problem solving skills have got very rusty, so I browsed deeper into the Flickr APIs.
I've been reading RESTful Web Services
Boy it was easy.
Now if you end up reading Leonard and Sam's book you'll discover that in their view, Flickr's REST API is not truly REST but actually a REST/RPC hybrid. He does go to great length to make the case for pure REST services but in my view, from a usability point of view, this ticked all of my boxes.
It's the above criteria that have been used to design the new Esendex API that will be coming to the Internet very soon. We are going REST.
The first release will be available in March to support an exciting new service we are bringing to market. This will be a sub 1.0 version, beta subset of the full API which we will augment to support the full feature set over the coming months.
We will of course formally announce the release on the Esendex Developer web site, and we will be maintaining support for our existing API.
Committing to this approach has also opened up some fascinating possibilities for our applications as well how we can build composite applications that combine all sorts of other services with our own. It's going to be a very RESTful year.
The LOCATE message is used by service clients to discover the existence and state of services on the network. Sent as a UDP broadcast it requests information about a specific service.
Upon receipt of a LOCATE, the relevant service is expected to respond immediately by broadcasting an ANNOUNCE message to enable the client to understand the current state of the service.
LOCATE [service name]
Where
| service name | A name for the service, well known within the system domain |
A typical LOCATE message would look like this.
LOCATE SMSCProxy
Use of the LOCATE message is optional by the client if the intervals between ANNOUNCE broadcasts by the service are sufficiently small. In many situations though, this delay is unacceptable and the client will initiate the ANNOUNCE broadcast by sending a LOCATE message.
Having both LOCATE and ANNOUNCE messages also has the side benefit of allowing the protocol to have a comedy name.
I've just been booking travel and accomodation for Web 2.0 Expo in San Francisco. I went to the European version last year in Berlin and it was a bit of an eye opener for me. The venue was pretty poor, both location and internal, but the subjects covered struck many chords with me and work I'm engaged in.
I've spent the last few years focussing on the mobile sector, eschewing Internet and developer conferences in favour of mobile specific affairs like Mobile World Congress and Global Messaging Congress. While these conferences have been fascinating and tedious to various degrees they have always been successful in giving me time to reflect and cogitate on our services, how they fit into the mobile ecosystem and what they should become in future.
Web 2.0 Expo in Berlin made me realise that there is a lot of amazing work being carried by the darlings of the Web 2.0 world that have many parallels with the challenges we face at Esendex. By immersing myself back in to the web world as well, there was a lot that I could learn.
As we continue to grow at the phenomenal rate we have been and evolve our services we come across all sorts of scaling and performance challenges. These definitely rank in the 'nice to have' category of problem but they're no less challenging. Hearing about lessons learned by the likes of Flickr et al. can only help us decide how we continue to stay a few steps ahead of demand in the future.
These new services have also upped the ante when it comes to what an Internet service should be able to provide. Their use of AJAX and REST architectures have provided a rich user experience far beyond where, to be brutally honest, we are right at the moment.
That is changing, we have some exciting new services coming to our customers over the first half of this year that bring us bang up to date. We've spent so much time focusing on making sure our messages get through in a timely fashion we've probably taken our eye off the ball on the front end usability. So there has been a big focus in the development team on making our services richer and easier to use. Obviously I'll fill you in as soon as I can.
I suspect that the Berlin conference did suffer from being smaller than it's American parent, so I'm looking forward to seeing how it works on a bigger scale. There are also worse places to go than San Francisco, not least to get my fix of unbridled, Bay-Side, can-be-done-ness.
The ANNOUNCE message is the core message in SLAP. Sent as a UDP broadcast, it announces information about a service to the network. It's purpose is to inform interested clients about the current state of the service.
ANNOUNCE [service name] [state (Green|Amber|Red|Blue)] [service uri] <[optional attributes (name=value)]>
Where
| service name | A name for the service, well known within the system domain |
| state | The current state of the service as represented by one of 4 values Green, Amber, Red or Blue, see below |
| service uri | Unique network address for the service. Can be any format that the client would recognise. |
| optional attributes | An optional series of attribute value pairs seperated with a new line. |
Service states are used in SLAP to give a simple indication as to the service state.
| State | Description |
|---|---|
| Green | Service is fully operational and available |
| Amber | Service is operational but under load |
| Red | Service is not operational |
| Blue | Service state is unknown |
If more information is required, then the service would include this with the optional attirbutes of the message. The content of these atttributes is not defined and is open to individual system implementations.
For example, information on current service performance (transations per second, queue size, memory utilisation, etc) could be included to allow the client to make utilisation decisions.
A typical ANNOUNCE message would look like this.
ANNOUNCE SMSCProxy Green node1.company.com:1234 Route=Operator1 Binds=3
Generally these messages would be broadcast at regular intervals to keep clients updated with service information. There are times when a client needs to actively interrogate the state of a service, for instance on startup, for which the LOCATE message is defined. More of that in the next post on SLAP.
Part of the reason I've been so quite on the blogging front of late is that I've got carried away with a bit of development spiking. Nothing that goes straight into production I hasten to add. I'm too far away from our development processes to be trusted with access to the live source tree, and anyway Nicholas (Development Manager) probably wouldn't let me.
I am in the enviable position of being able to create proptotypes, test things out and hand them over to the professionals to make them a reality. It's one of these projects that became SLAP (Service Location and Announcement Protocol).
One of the key technical challenges facing an organisation like ours is scaling our applications out to meet demand while at the same time keeping everything highly available.
Our architecure comprises a series of software agents that pass messages between each other routing and manipulating traffic based on a series of rules. For example
As I'm sure you can imagine running multiple instances of these agents across multiple servers and have them know the state of the agents around them so messages get passed around reliably is no mean feat. Should an Mobile Operator connection become unavailable or we lose a server, the agents should be able to adapt to that.
One option is to use hardware load balancers, as we do in front of our web servers, but this quickly becomes very expensive and very costly to maintain as you end up with multiple layers of load balancing. Also it's very hard to dynamically configure the network of agents to cope with peak periods of traffic.
After reading Scalable Internet Architectures
Nodes broadcast one of 2 message types: ANNOUNCE and LOCATE over UDP which are subscribed to by other nodes on the network. These nodes can be application monitoring agents or clients wishing to consume services offered by the node.
It's simplicity hides a power that has allowed us to architect our system to self configure and repair itself in the event of failures. In subsequent posts I'll describe the specifics of the protocol and our describe our experiences using it.
I was interviewed along on with BBC Radio Nottingham with Huw Hilditch Roberts of the National Farmers Union about the way they are using our Web SMS to keep farmers up to date on the latest about Bluetongue and Foot and Mouth disease.
It was my first time live on the radio, slightly nerve wracking but kinda fun at the same time. Alan Clifford was very kind, he certainly didn't take a John Humphries or Jeremy Paxman approach.
If you want to check it out go to BBC Radio Nottingham listen again and choose DriveTime on Friday and fast forward to 40 mins in.
Next stop Newsnight ;-)
I realised over the weekend that I'd been a bit quiet on the blog front of late. We're going through some big changes here at Esendex towers and it's been more a symptom of focusing on those.
We're a profitable, successful organisation with an ever growing customer base in 5 countries around the world. This gives a fantastic platform to develop the business further into new markets and new levels of service. Esendex 2.0.
We're currently focusing our efforts in 3 areas:
We are launching a new web site very soon. We've integrated with the Episerver CMS to enable rapid updates and changes to the content.
We've always been praised on the quality of our customer service and support and as we grow we're looking at more scalable ways of offering that same level of service. One of the main improvements will be the inclusion of blogs run by the Development team and Support team as well as a broaded corporate blog.
Both the Developer and Support sites are being managed directly by the relevant teams and the blogs will be their way of the keeping our customers up to date with everything they need to know.
In the case of the Development this will be posts on using the SDK, new features we're adding to support developers and a more in depth explanation of the approach, rationale, etc.
The Support bog will be broader in it's appeal, giving customers information on releases, new features, how tos. planned outages and anything else the team feels would assist customers.
A lot of work has been going on to develop our system architecture to support a new range of services as well as our forecasted growth over the next 5 years. While the system is more than capable of coping with our current levels of traffic, the last thing we want is technology to be the constraint on our aspirations.
I will be posting about some specific developments over the coming weeks. Essentially we are moving our platform from an Enterprise to a Mobile Operator architecture. The two key challenges we face:
It has been a real challenge but the results are looking very exciting in a nerdy, check out that performance graph, kind of way.
Over the coming months you'll be seeing a range of new services. Can't go into too much detail but a new era is dawning.
Ian and Jonathon have been paired up on resolving a long standing omission from our system, aggregating inbound long messages.
Currently when a subscriber sends a long message to one of our SMS long numbers or short codes, each part of that message arrives in the inbox, or is pushed to our customers web service, individually. Aggregating the messages for multiple accounts while maintaining performance was a reasonably significant architectural problem to resolve.
We had looked at it before but were never quite happy with the solution. I finally committed the development time to do it as a result of the a case being raised by a very patient and reasonable customer. This time we were ready.
One of the constraints on any changes we make to our systems is preserving the interface to customers who use our API. In our initial plans we were going to potentially break it for some people.
Ian describes it in more detail in his post: Esendex Inbound Multipart Messaging Update. While we weren't strictly breaking the interface, if our customers had made reasonable assumptions about the length strings being passed it could break their application.
Not happy with this, the three of us sat down and we did come up with a solution that satisfied the internal architecture of our system while still supporting our existing customers.
We are going to provide an API option on an account to aggregate long messages. While we'll be storing messages in their aggregated form in our system. For those accounts with that option switched off, the API will re-split the messages before delivering them.
All existing accounts will have the option turned off by default and all new accounts will have the option turned on.
It makes the Options page a little more complicated but it keeps everyone working, which is what it's all about.
Summer is traditionally a quite time in many businesses, especially in France and Spain. Phones ring less frequently, inboxes are manageable, Out of Office replies are more prevalent and desks are empty as people enjoy holidays.
Normally this would be accompanied by a drop in traffic. Less people working, less to send. Not so this year, in the UK and Ireland at any rate, our traffic has barely dropped off.
I've excluded Australia from these observations because of the seasonal dfferences. They're deep in winter so must be getting loads done.
As Julian posted in Bad News is Good For Business!, we've had our fair share of extreme circumstance in the UK this summer which does drive usage for our customers. But the general base load of traffic is pretty consistent with previous months perhaps with just a slight reduction in the growth rate we normally experience.
I wonder if this is another indication of how entrenched SMS is becoming in the business communication mix. Business continues through the summer so communication continues. Where previously certain sectors were early adopters of the technology, a broader profile of business types now use SMS, ironing out those seasonal differences.
Alternatively, we've experienced such stellar growth in the lead up to the summer that this has swamped any seasonal variations we would expect to see. I guess we'll find out in September.
One of the downsides of running Premium SMS services for customers is having to deal with irate subscribers who feel, rightly or wrongly, that they have been charged for services that they didn't subscribe to.
When a subscriber has an issue, they generally call their network operator who calls up the details of the service provider who run services on a given shortcode. If they get this right, not always by any means, the subscriber calls us.
We, as a service provider, have a dedicated number for these and can quickly cancel any subscriptions and give the subscriber the details of our customer so they can approach them for a refund.
All this is fine if the subscriber acts reasonably, this isn't always the case.
It never ceases to amaze me how people deem it acceptable to scream abuse at people over the phone as if that is going to help expediate matters.
My team are on the front line and no matter how much I reassure them that's it's perfectly acceptable to hang-up on people who act this way, they are still affected by it.
It's grossly unfair that some reactionary little prat with anger management issues can upset and offend people at will with no comeback on them.
It does come with the territory but I wish these people would learn to act like adults. The world would be a better place for them and the rest of us.
Talking to one of our network account managers recently, they told me that we were their best customer of SMS Long Numbers. I was a initially surprised but a quick search of other SMS providers sites reveals very few companies are promoting these heavily to their customers.
We've always believed that for SMS to be adopted as a business communication tool it had to provide a reply path. You can reply to emails, phone calls, faxes why not SMS. It's a core part of our offering. So much so in fact if you sign up for a trial, we'll give you one to try out.
The conventional approach is to use short-codes for this. Short, memorable numbers that are great for one time responses to ads or other marketing efforts. The problems with short-codes however are:
Now we've got round some of those issues by use of keywords to allow routing of inbound messages to the correct account holder. While this is great for single response channels, it's un-workable for conversational communication. You can't expect users to remember to prefix every message in a conversation with a keyword.
Enter the long number:
When it comes to charging not all numbers are created equal. Network providers in the Channel Islands (Cable & Wireless, Jersey Telecom) and on the Isle of Man (Manx Telecom) have numbers that are in the standard UK range, ie starting +44 (0)7.
Unfortunately however, the interconnection rates between these companies and the mainland UK operators (Vodafone, Orange, etc) is not regulated by Ofcom. This means that they can charge what they like and in T-Mobile's case they do.
T-Mobile regard messages to these providers as travelling to Europe Zone 1 and therefore treat them as an international messages rather than national. Which means, if you're a T-mobile subscriber, it doesn't come out of your plan
A quick peruse of the Ofcom's mobile number range allocations will tell you very quickly if the number your service provider is offering you will be affected.
In short if your number starts with 07786 or 07800 you're going to be OK, otherwise you need to beware.
At Esendex we provide numbers that are in the regulated number range in all of the countries we operate in. In our view there should be no difference between texting into a service or texting someone's mobile. Remove the silly barriers and users will adopt and embrace these services.
Ofcom have started a Wholesale SMS Termination Market Review. I have no doubt it will operate at glacial speeds but it could be just what the industry needs to remove this confusion.
I'm completely hooked on Google Analytics or rather the information it gives me about how our web site is being used. One of my favourite information sets is the referring web sites. Where did people click through to our web site from.
This throws up some occasional gems where real people are developing against our API and sharing it with their peers.
One that cropped up recently was a plug-in for Nagios an Open Source host, service and network monitoring program. One of the users as developed a Nagios plug-in: SMS HTTP Gateway - Esendex that sends system alerts using our SMS API.
Another example is EsendexSMS, a plugin for the SugarCRM open source CRM system. We actually managed to track this one down to one of our Australian customers.
The power of Internet service delivery is demonstrated time and time again by these and similar applications. Nice to be a part of it.
The Development team put up a beta version of a web service to support the new Outlook Mobile Service that ships in Outlook 2007.
After all the faff of setting up Outlook plug-ins and downloading software, etc this seemes like a great way of SMS enabling Outlook. It's fully integrated and just requires an Internet connection to operate.
If you want to try our service, you create a new email account of type 'Outlook Mobile Service' and set the Web Service Address to: https://www.esendex.com/api/oms/omsservice.asmx.
The username is in the standard Esendex structure for email integration:
<Esendex Account Referrence>\<Esendex Username> eg: EX0123456\user@company.com
I found a full restart of Outlook (by logging off and on to my Windows account) was required to get the service properly configured. But after that, it worked a treat.
Currently this service is Outbound only. Replies will still come back into you Inbox with email forwarding but Outlook won't recognise them as SMS messages until we make some changes to our Email forwarding logic. Due for release at the end of August.
Keen to get feedback on this so please contact support@esendex.com or leave comments here with any thoughts, experiences, suggestions.
One of the key challenges faced by our sales team in Australia is selling on the benefits of reliability and availability. It seems there are still a lot of people who consider these generally less important when choosing and SMS service provider, price is what it's all about.
The UK market was like this a couple of years ago. But we've seen a definite shift towards customers using selection criteria that include reliability and availability more recently. Notably several customers who left us for a keener priced service and come back finding it didn't meet their requirements.
In these early days, price was everything it was about proving business cases and trying out ideas. This isn't critical to my business, why do I need to pay for a UK routed service when I can buy it more cheaply off-shore.
These services typically used off-shore routes taking advantage of lax interconnect policies to submit messages to subscribers on the UK operators without paying interconnect fees (3p between UK operators and possibly more internationally). The UK operators soon started closing these routes with AA19 Interconnect Agreements, effectively pricing the off-shore routes out of the market.
As the price went up, people started to notice the intermittant nature, or total lack, of delivery receipts.
The non-GMT timestamps (the time the message is marked as sent is that of the sending SMSC and not the destination) became an irritating source of confusion.
The availability of these systems were constraining business processes that had come to rely on the delivery and/or receipt of SMS messages.
We've always built our service around the best quality routes for send and receiving messages ensuring delivery receipts are available, timestamps are correct and the system is available. It did take a while for the the UK market to realise that we had a point.
Recently in Australia, Telstra seemed to have blocked one of the key offshore routes in used by many of the budget SMS providers.
Most likely this was due to high levels of spam. Cheap routes make it economic for SMS spam to be carried. The problem for legitimate traffic also using this route is that the operator will just block everything, irrespective of content. Another way availability can suffer.
This kind of incident is obviously great news for Esendex and companies like us who provide quality services. Telstra is educating the market about the true cost of using off-shore routes far more effectively than any sales pitch from us.
Not everyone can justify a quality service straight away, back to trialling, proving business cases, etc. The problem is they come to rely on the service more quickly than they expect and when it goes wrong, things get messy.
Kevin, our Systems Manager, has been evaluating some interesting technology that should not only make his job easier, but also save some power as well.
VMware provide arange of server virtualisation products that allow you to run multiple operating system instances on one instance of hardware.
There are number of servers sitting in our production network performing essential roles (Windows Domain Control, DNS Services, monitoring, etc) that do not result in heavy hardware utilisation. From a utilisation point of view the services could easily run on one server. The problem with these services however, is they're not necessarily good bed fellows and are recommended to run on separate boxes. These services also need to be resilient so 2 servers are required for each service.
Enter VMware, their technology sits between the hardware and the operating system unlike Windows Virtual Server which is an operating system service. This means any OS can be installed side by side, ideal in our environment where we do have some Linux servers in place.
If the technology passes our evaluation, 6 servers that were running 3 services in a resiliant architecture can become 2. 3 virtual servers on each box each 'side' running a copy of the other.
So we need less hardware which also means less power (see Power is the currency). Instead of 6 servers consuming power just to sit around and do very little. 2 servers are running optimally, reducing the consumption by at least 50%.
This is something I've been thinking about for a while and follows on from a post Julian made: Conduct Unbecoming of a Gentleman
I'm a reader of SMSTextNews as is Steve Procter of iTagg. It's not unheard of for him to mention in his comments how iTagg are a wonderful company that follow the rules and make sure subscribers of the services they host don't get ripped off.
Now, I'll be honest, I used to think, 'give it rest Steve, being sanctimonious isn't endearing'.
I've changed my mind.
At Esendex, we work from a premise too of doing all the right things. Like enforcing compliance on our customers to ensure consumers don't get ripped off. Like, not bare-faced lying about other companies as described in Julian's post.
I think it's time to shout from the rooftops about being responsible, acting appropriately and distancing ourselves from the companies facing allegations surrounding the GMTV debacle and all the other sordid stories.
Companies who want a service for the long-term need to know that the service provider is going to be around for the long-term. Flounting the regulations and acting irresponsibly/dishonestly means the service provider is at risk of being heavily punished. That punishment, and/or the ramifications, can destroy a company.
Now that approach may suit some key individuals who have made their money and are happy to walk away. But what about their customers who are left without the service they've come to rely on. For so many of our customers, our SMS service is integral to their business processes. Losing it would cost them an awful lot of money.
Now I'm sure some of you are reading this and thinking what a load of wishy-washy, liberal nonsense and heh perhaps even a little sanctimonious towards the end. Maybe, that I should stop being naive, business is war, any weapon, any cost, winning is all that matters.
Ultimately the market will decided and we'll see who's left standing at the final reckoning.
Our flag is raised, we've shown our colours, now bring it on.
