Showing posts with label Development. Show all posts
Showing posts with label Development. Show all posts

Wednesday, 2 July 2008

Judging is complete

Finished the judging for the Esendex Developer Challenge today. A tricky exercise, the entries were diverse, in the end however we were all in agreement.

The winners are now safely tucked away into in a metaphorical gold envelope. As soon as I've spoken to them, we'll announce it to the world.

Tuesday, 10 June 2008

Security in anonymity

Amazon have had some recent outages (Bots to blame for Amazon.com outages? | The Register) which raise a spectre of doubt for me over the viability of cloud computing services.

These services allow smaller organisations to benefit from large investments in computing power made by a small number of organisations, Amazon, Google, etc and buy IT infrastructure as a service.

The concern for me is that they provide a very, very big and obvious target for people of nefarious intent to aim at. Global brands like Amazon represent a prize for hackers and it would seem if one goes down, everyone goes down. Their being S3 services were affected as well, taking down all customers services that rely on them.

Doing it for yourself, on your own servers gives you the protection of the crowd in much the same way as being in a shoal of fish provides protection to the individual members from predators. Of course, you have to do a certain amount of self protection but being far less likely to be a target makes that a lot cheaper.

The counter argument is that these large cloud services organisations have the resources to hire the best defensive minds in the business to protect their investment, a level you couldn't possible afford or justify on your own. The problem is that these people are defending against the best attacking minds in the business because they're protecting the biggest prize, it's a classic arms race.

Another argument is that these services allow you to scale your service rapidly in the face of unprecedented demand. You never know, your service might become de rigueur and in the Internet world that means millions of hits.'You only get one chance'.

Entrepreneurs lap this one up, being the eternal optimists that we have to be to survive the start-up trials, of course my service is 'the one' I'm just waiting for the market to realise and then I'll be ready. Looking at it a little more pragmatically, this is a million to one shot.

What's more likely, should your business succeed, is that growth may well accelerate but not more than is copable with. The trick is to architect your service so you can rapidly scale it, then it just becomes an issue of cash hardware. If you've got your business model straight then this shouldn't be a problem.

So, home grown for me. It's not hard or expensive these days with so many open source services and stiff competition in the ISP market; and I haven't even got onto the perils of locking your pride and joy to a proprietary app hosting architecture.

Friday, 28 March 2008

iPhone goes for the enterprise, BlackBerry fight back

It's an iPhone with a QWERTY keyboard!

BlackBerry 9000 in the wild

OK so no touch screen interface but this promises to be a serious bit of kit that will cement BlackBerries in the enterprise. Must check out the BlackBerry development kit so how that stacks up against the iPhone SDK that Jonathan is wrestling with at the moment.

Code Comments are they really pungent?

XP tells us that comments are a bad coding smell indicating that some functionality should be refactored out into a method. I've never been totally comfortable with this and Matt Stephens as posted a very good article as to why.

Comment judiciously, refactor if needed, avoid the 'f' word

Tuesday, 25 March 2008

Are Symbian and J2ME dead?

The launch of the iPhone and Android SDKs represent a huge leap forward in the opportunities for mobile device application development. The question for me is how do the current incumbents, Symbian and J2ME, react.

I've talked before about my continued disappointment with mobile applications (Mobile Phone Applications, when will they ever take off?). These experiences have been very much routed in the pre iPhone/Android era.

In the Symbian/J2ME world, applications are poorly integrated with the phone, hidden by layers of menus and generally underwhelming. The world according to iPhone/Android has the opportunity for the application to be front and centre, and in Android's case, closely integrated with the phone functions.

The second half of 2008 promises to be a fantastically exciting time for mobile device users. The iPhone has redefined the mobile device and the version 2.0 firmware with its enterprise features will deliver a device that kicks some serious Windows Mobile butt. Can't believe they're on version 6 and it's still an overweight, clunky to use, buggy monstrosity.

Now apps for the iPhone are not going to be without issues. The big one for me is the inability/refusal to allow background processes to run. It stops all sorts of useful features like background data updates and push facilities that would really make it a BlackBerry killer. That said, it's an amazing user interface experience, and as it continues to demonstrate, even without 3G data transfer speeds, iPhones are blazing a trail in mobile web usage. Though I do suspect most of this is on home WiFi networks.

The killer feature for me is the remote update capabilities that are built into the OS. The link to iTunes is critical to the devices operation and this allows Apple to push new updates, including strategy changes, at will. I can't remember the last time I updated the firmware on a Symbian phone.

Android is taking a very different line, an almost completely open platform. Full access to all the device's functions. Now this is really a desktop-like development experience. No-holds barred, access to everything, only constrained by your imagination type environment. Only one thing is missing, handsets. It seems Samsung and HTC are in a race to deliver the first but apparently this won't be until the end of this year! So develop what you like, it 'aint going to be on a device until 2009.

There seems to be a huge opportunity for Nokia to leverage their number 1, by a long way, position in the world handset market and bring something truly powerful and pervasive to the market. Unfortunately it seemed their response was to make it even harder to get Symbian apps on phones. Why Symbian Signed must die gives a good account.

Symbian can push out news stories about how many handsets ship with Symbian, 77.3m Symbian smartphones shipped in 2007 but for me that's 77m handsets that aren't going to run Symbian apps, it's just too difficult for people, especially enterprises.

Apple are treading a very delicate path with the iPhone, keep it exclusive but get it adopted by enterprises. Historically their products haven't been adopted by this segment. If their aspirations are to get the iPhone into the hands of key personnel and in turn those personnel buy a Mac for home instead of a Windows PC then that's probably a huge result.

I'm pretty neutral on Android. Yes it's flexible, yes it's open, but it's a while before before people can actually use it and who knows what will happen then. I do wonder whether there is a bigger play here involving GrandCentral a mobile device, closely integrated with that service could be very disruptive indeed.

Symbian, IMHO, have lost the plot already. They have huge penetration but low usage and no established update process. It's going to take something dramatic from them to enable them to maintain their dominance of the high-end mobile device market.

Tuesday, 11 March 2008

Putting SLAP to use - Dynamic Config

Now I've got the specifications of the SLAP protocol out of the way, I'd like to describe how we're using it and the benefits we are seeing. The first being the dynamic configuration of service end points.

In a service like ours we have a series of nodes that process messages and apply rules to manipulate and route them. If we want to change the location for a service, we have to first notify all the upstream services of the change of location.

In a planned scenario this can be laborious. All services that may make use of the service have to be reconfigured, often requiring a restart. It is possible to have these services monitor a config file for changes to avoid a restart but this doesn't remove the need to manually go through each service config and make the necessary changes. This becomes increasingly complex in a distributed system spanning many machines and networks.

The unplanned scenario is even less desirable. There are many reasons why a service can stop and this will generally be at the most inopportune times. Having to manually reconfigure services in response to a failure is not acceptable in a high availability scenario like ours.

The other problem with the unplanned scenario is that the client service has to deal with the unavailability at the point it's attempting to consume the service. Say for example this is a network service, it could be waiting for a network timeout before it recognises that the service is not accepting requests. It then has to fire-fight, cleaning up because it thinks the service is unavailable.

Far better for it to be told categorically that the service is unavailable, it can then queue requests or whatever has been coded as appropriate, while it waits to be informed that the service is operational again.

It is possible to use separate load-balancers to handle this kind of outage, but they cost money, need configuring, draw power and the solution the bring is by no means dynamic. I'll actually discuss load-balancing using SLAP in a subsequent post.

In the SLAP world, services are not configured to use fix endpoints like sip:ems3.prod1.esendex.com:8067 but rather are bound to well known service names eg: slap:smsrouter. The actual endpoints to the service are advertised in ANNOUNCE in order that clients can maintain a record of the current state of the services they need to consume.

If a client service wants to make use of an smsrouter service, it checks the state in it's local service state table before sending the request to the correct service URI. This information is kept up to date by the service. Most will announce their state on a periodic basis but I'd also consider it good practice for the service to send an ANNOUNCE when it's state changes, perhaps when it's under load or is shutting dowm, to ensure all clients are kept up to date.

The development team at Esendex have also found it very useful when building and debugging services as Jonathan describes in SLAP, my service’s up!. This was an unforeseen benefit but another one that has cut out a lot of the hassle with debugging services. The guys can step through the code, find the issue, write the unit test and rebuild very quickly which get's us to market far quicker.

More on load balancing soon.

Monday, 18 February 2008

A very RESTful experience

In May this year, a group of us (parents, friends and teachers) from my sons' school are cycling from Nottingham to Paris in order to raise money for the school. My role in this exercise, in addition to cycling, is to build and manage a website (www.r4rh.org) to promote the event.

Part of that web site includes retrieving a set of latest photos from Flickr tagged r4rh or RideForRoundHill. I've built the site using ASP.NET so naturally downloaded the Flickr .Net API Kit. My needs were very simple, just searching for a list of photos, and it worked brillantly until I published the site.

For some reason, I don't know why, the reference to the FlickrNet.dll didn't see, to carry across. Initial investigations didn't uncover a solution, one of the problems I find with doing this kind of thing these days is that my integration problem solving skills have got very rusty, so I browsed deeper into the Flickr APIs.

I've been reading RESTful Web Services by Leonard Richardson and Sam Ruby recently and the ethos of REST style web services has been intriging me greatly so I decided to have a go at consuming the Flickr REST API.

Boy it was easy.

  • Do GET from URL with API key and search parameters
  • Receive XML document
  • Parse XML document to get list of photos
  • Build list of images to render on web page.

Now if you end up reading Leonard and Sam's book you'll discover that in their view, Flickr's REST API is not truly REST but actually a REST/RPC hybrid. He does go to great length to make the case for pure REST services but in my view, from a usability point of view, this ticked all of my boxes.

  • Simple to make the API call: just construct a URI
  • Simple to interpret the results: wide support among development platforms for XML parsing
  • Loosely coupled: Flickr can add new data to the output without breaking my code

It's the above criteria that have been used to design the new Esendex API that will be coming to the Internet very soon. We are going REST.

The first release will be available in March to support an exciting new service we are bringing to market. This will be a sub 1.0 version, beta subset of the full API which we will augment to support the full feature set over the coming months.

We will of course formally announce the release on the Esendex Developer web site, and we will be maintaining support for our existing API.

Committing to this approach has also opened up some fascinating possibilities for our applications as well how we can build composite applications that combine all sorts of other services with our own. It's going to be a very RESTful year.

Wednesday, 30 January 2008

SLAP - LOCATE Message

The LOCATE message is used by service clients to discover the existence and state of services on the network. Sent as a UDP broadcast it requests information about a specific service.

Upon receipt of a LOCATE, the relevant service is expected to respond immediately by broadcasting an ANNOUNCE message to enable the client to understand the current state of the service.

LOCATE [service name]

Where

service nameA name for the service, well known within the system domain

A typical LOCATE message would look like this.

LOCATE SMSCProxy 

Use of the LOCATE message is optional by the client if the intervals between ANNOUNCE broadcasts by the service are sufficiently small. In many situations though, this delay is unacceptable and the client will initiate the ANNOUNCE broadcast by sending a LOCATE message.

Having both LOCATE and ANNOUNCE messages also has the side benefit of allowing the protocol to have a comedy name.

Monday, 28 January 2008

Web 2.0 Expo, swapping Berlin for the Bay Area

I've just been booking travel and accomodation for Web 2.0 Expo in San Francisco. I went to the European version last year in Berlin and it was a bit of an eye opener for me. The venue was pretty poor, both location and internal, but the subjects covered struck many chords with me and work I'm engaged in.

I've spent the last few years focussing on the mobile sector, eschewing Internet and developer conferences in favour of mobile specific affairs like Mobile World Congress and Global Messaging Congress. While these conferences have been fascinating and tedious to various degrees they have always been successful in giving me time to reflect and cogitate on our services, how they fit into the mobile ecosystem and what they should become in future.

Web 2.0 Expo in Berlin made me realise that there is a lot of amazing work being carried by the darlings of the Web 2.0 world that have many parallels with the challenges we face at Esendex. By immersing myself back in to the web world as well, there was a lot that I could learn.

As we continue to grow at the phenomenal rate we have been and evolve our services we come across all sorts of scaling and performance challenges. These definitely rank in the 'nice to have' category of problem but they're no less challenging. Hearing about lessons learned by the likes of Flickr et al. can only help us decide how we continue to stay a few steps ahead of demand in the future.

These new services have also upped the ante when it comes to what an Internet service should be able to provide. Their use of AJAX and REST architectures have provided a rich user experience far beyond where, to be brutally honest, we are right at the moment.

That is changing, we have some exciting new services coming to our customers over the first half of this year that bring us bang up to date. We've spent so much time focusing on making sure our messages get through in a timely fashion we've probably taken our eye off the ball on the front end usability. So there has been a big focus in the development team on making our services richer and easier to use. Obviously I'll fill you in as soon as I can.

I suspect that the Berlin conference did suffer from being smaller than it's American parent, so I'm looking forward to seeing how it works on a bigger scale. There are also worse places to go than San Francisco, not least to get my fix of unbridled, Bay-Side, can-be-done-ness.

Friday, 25 January 2008

SLAP - ANNOUNCE Message

The ANNOUNCE message is the core message in SLAP. Sent as a UDP broadcast, it announces information about a service to the network. It's purpose is to inform interested clients about the current state of the service.

ANNOUNCE [service name] [state (Green|Amber|Red|Blue)] [service uri]
<[optional attributes (name=value)]>

Where

service nameA name for the service, well known within the system domain
stateThe current state of the service as represented by one of 4 values Green, Amber, Red or Blue, see below
service uriUnique network address for the service. Can be any format that the client would recognise.
optional attributesAn optional series of attribute value pairs seperated with a new line.

Service States

Service states are used in SLAP to give a simple indication as to the service state.

StateDescription
GreenService is fully operational and available
AmberService is operational but under load
RedService is not operational
BlueService state is unknown

If more information is required, then the service would include this with the optional attirbutes of the message. The content of these atttributes is not defined and is open to individual system implementations.

For example, information on current service performance (transations per second, queue size, memory utilisation, etc) could be included to allow the client to make utilisation decisions.

A typical ANNOUNCE message would look like this.

ANNOUNCE SMSCProxy Green node1.company.com:1234
Route=Operator1
Binds=3

Generally these messages would be broadcast at regular intervals to keep clients updated with service information. There are times when a client needs to actively interrogate the state of a service, for instance on startup, for which the LOCATE message is defined. More of that in the next post on SLAP.

Monday, 21 January 2008

SLAP, an introduction

Part of the reason I've been so quite on the blogging front of late is that I've got carried away with a bit of development spiking. Nothing that goes straight into production I hasten to add. I'm too far away from our development processes to be trusted with access to the live source tree, and anyway Nicholas (Development Manager) probably wouldn't let me.

I am in the enviable position of being able to create proptotypes, test things out and hand them over to the professionals to make them a reality. It's one of these projects that became SLAP (Service Location and Announcement Protocol).

One of the key technical challenges facing an organisation like ours is scaling our applications out to meet demand while at the same time keeping everything highly available.

Our architecure comprises a series of software agents that pass messages between each other routing and manipulating traffic based on a series of rules. For example

  • A outbound bulk SMS will head straight to an SMS Router for it to decide based on the state of a set of Mobile Carrier connections which is the best current route.
  • A premimum SMS message will be passed to a Subscription checking agent to confirm that the subscriber is valid, and then possibly onto a network lookup agent to resolve the destination network before heading onto the SMS Router for onward submission via the correct network for billing purposes.

As I'm sure you can imagine running multiple instances of these agents across multiple servers and have them know the state of the agents around them so messages get passed around reliably is no mean feat. Should an Mobile Operator connection become unavailable or we lose a server, the agents should be able to adapt to that.

One option is to use hardware load balancers, as we do in front of our web servers, but this quickly becomes very expensive and very costly to maintain as you end up with multiple layers of load balancing. Also it's very hard to dynamically configure the network of agents to cope with peak periods of traffic.

After reading Scalable Internet Architectures I was inspired to come up with a simpler, and cheaper, solution. SLAP was born which is an alternative, software approach that is deliverying huge benefits to us already.

Nodes broadcast one of 2 message types: ANNOUNCE and LOCATE over UDP which are subscribed to by other nodes on the network. These nodes can be application monitoring agents or clients wishing to consume services offered by the node.

It's simplicity hides a power that has allowed us to architect our system to self configure and repair itself in the event of failures. In subsequent posts I'll describe the specifics of the protocol and our describe our experiences using it.

Friday, 18 January 2008

Still loving my iPhone

I'm pretty notorious for having a very short attention span. I'll jump passionately in to a new service or toy with both feet, love it for a couple of weeks until something else takes my fancy. Julian, as you can see from his derisory comments on : iPhone - Smartphone for the Normob?, has been of the opinion that my iPhone purchase would go the same way.

Well I'm still loving it and the latest updates announced at a MacWorld have made it better. It really is a device that fits into my personal life.

Yes the camera's not great and the data coverage can be irritating but it's so well thought out and put together that I still keep coming back for more. I've even got used to the keyboard.

This year will see a slew of copies from Nokia, LG, Samsung, et al and it'll be really interesting to see how these manifest themselves. I think the challenge for them will be to not try make it backwardly compatible, interface wise, with the rest of their estate.

One of the key reasons the iPhone is such a success is that Apple have taken decisions about how you want to use the device on your behalf. Their team of UI designers have worked out the best way for it to work so you don't have to.

The temptation when developing software is to give user full control, lots of features that they can take advantage of. You never can tell what a user might want to do so make sure you've got your bases covered. The danger with this approach is that you end up with a sea of options that confuse the user rather than empower them.

I liken this to the multiple camera options you get (at least I assume you still do) with Sky Sports coverage. So you can watch Wayne Rooney scratch is rear or John Terry yelling at his team mates while the rest of the game is going on.

This kind of service doesn't appeal to me at all. Why should I make decisions about the best angle when a highly paid (assumption) highly experienced sports TV producer is there to make the decisions for me and make sure I don't miss the best bits.

The iPhone has been a liberating experience for me. Usually with a new device I'm desperate to dive in and configure, load things, try things out and I just end up getting frustrated and ultimately fall out of love with it. Apple's designers have prevented me from doing that while making some great decisions about how the iPhone should be used.

This time I'm using the device in the way it was designed and our relationship is stronger than ever.

Tuesday, 13 November 2007

Not thinking, on a budget

One of my post Web 2.0 Expo book purchases was Don't Make Me Think by Steve Krug. I obviously missed the boat as this is now the second edition published in 2005 but it cropped up on one of James Kalbach's slides so I thought I'd check it out. 48 reviews on Amazon with an average 5 star rating was enough for me.

It's everything a business book should be. Short, I've read it in two evenings, and with all of its points clearly made.

There is some really valuable material in here, especially the chapters on how to do useability testing on a budget. I'll definitely be getting the video camera out when I get back to the office.

Having just gone through the process of designing and developing the new Esendex web site I feel that perhaps we've missed a trick.

In reality however we've made some massive improvements and web site design and development should be an iterative process, the trees really do get in the way after a while.

Monday, 12 November 2007

Is Pair Programming really working for us?

Nicholas and I are having some misgivings at the moment about whether Pair Programming is all it's cracked up to be. We're not sure whether we're in danger of becoming dogmatic about enforcing it.

One problem is that in our environment requirements and demands on time can be quite fluid, especially as the development team also help our SMS API customers with their integration. This inevitably leads to pairs splitting and less optimal work sessions.

Another is that people are not created equal, there is inevitably a leader in any pair. This can be down to having more development experience, more experience of the part of the system being worked on or just having a more forceful personality.

This translates into a less than optimal contribution from one member of the pair. Especially in the latter case, the less forceful person is likely not to be contributing anyway near to their potential if they're being 'bulldozed' by the stronger party. This isn't necessarily down to arrogance or any malcious intent, it's just they way people are.

So what's the alternative?

We're going to trial working in pairs but on coding two tasks, ideally related to each other, individually. That way you have someone available, on your wavelength to discuss design decisions, patterns, tests, etc but you can knuckle down and get some tests written and functionality produced at your own pace. When you're ready, then you bring in your pair for a review.

This stays true to the concepts of discussion, collaboration and while the code reviews are notquite inline, they are very close to it.

We're running this as an experiment until Christmas. We'll report back

Thursday, 8 November 2007

The Space Tag Continuum

I've been at Web2Expo the last couple of days and at a session on Designing Tag based Navigation I got involved in a discussion about multi-word tags and how they should be input.

There are many sites out there, Flikr,tag cloud shown below, being a prime example, where they only support multi-word tags if you enclose in double quotes. Surely this is just poor design. It's a requirement enforced by a development decision rather than actually giving thought to how a user thinks.

Tags are used to categorise artificts, be they pictures, blog posts, documents or whatever. When you want to categorise something, often a phrase is best, eg: New York, New Years Party 2007, or Dan's Birthday.

Amazingly, to me anyway, people in the audience thought it was perfectly reasonable for the user to be required to either remove the spaces: NewYearsParty2007; or use double quotes: "News Years Party 2007". You'll notice above New and York appear has separate tags while some people have adapted their entry to remove the spaces for newyork.

To the first one I say there is no such word as NewYearsParty2007 and to the second one I'm not quoting anyone. We forcing the users to adapt their natural behaviour to fit a system that could just be designed better.

When we write lists, in English at any rate, we delineate with commas. What's wrong with doing that when inputing tags? Even better make it culture specifc.

I've posted previously about Normobs, I wonder if we need the concept of Norwebs (Normal Web Users). For those in the UK, I know it's also the name of a utility company but I don't think there's a trademark collision.

While it's easy to get excited about the possibilities of the latest web applications for them to realise their true potentional, we must remember the Norweb, for they are many and they are right ;-)

Tuesday, 9 October 2007

The problem of dependency

Regular readers will know that we've just launched our new web site. There are some teething issues, in certain, thankfully infrequently, situations the CMS system we're using doesn't like presenting the page to the site visitor.

My team are in close contact with EPiServer to get it sorted but it's leaving us feeling a little out of control. It doesn't sit well with the control freakery I've cultivated.

One of the key tenets of the Esendex Messaging Systems that has developed over the years is doing it ourselves. We've developed, much to the surprise of many people we interconnect with, our own protocol implementations to interact with their systems. On top of that are our own messaging systems, underpinning our customer applications. Our mantra has become:

DIY to get it right

I appreciate this might sound a little like I'm blowing our own trumpet and you're probably right. We're actually very proud of what we've done and it's given us a strong reliable platform that our customers can rely on.

Enter the third party CMS system. We're using it because it is very functional, does far more than our in house system we had built, and gives us the control we need. However when there is a problem we're paralysed. Frantically trying to debug the problem on behalf the supplier to try and sort the problem. It just doesn't sit right with us.

I have no doubt this was the right decision. The EPiServer platform is a cut above anything else we evaluated and gives us so many more features. We've just got to accept that sometimes we can't fix it.

Saturday, 29 September 2007

Protocol Therapy

An entrepreneur is required to wear many hats. In my case that can be anything from reviewing operational performance, developing new marketing channels, discussing finances with the rest of the board, to even sometimes sales.

For me this is part of the appeal of what I do. I'm better at starting things than I am at finishing so moving between different demands on my time keeps my work fresh and exciting. Though like everyone, I have to make sure there aren't too many plates spinning.

Recently however, I've been making a bit more time for the technician in me and doing some development. I've been investigating various services and protocols, building test systems to assess their potential to be offered as new services for our clients.

I've discovered that I find protocol implementation quite therapeutic. Give me an RFC, Visual Studio and a copy of NUnit and I can zone out for several hours.

Unit testing was made for this kind of development. I get lost in the mesmeric cycle of

  1. Identify requirement in RFC
  2. Write test for requirement
  3. Write code to make test pass
  4. repeat

The outcome is a robust implementation that meets the spec. as well as a general sense of well being.

Obviously I have to ration this kind of therapy, I have got a company to run, but it shows that relaxation can be found in the most unlikely of places as entirely personal.

To find mine, I just had to embrace my inner geek.

Saturday, 1 September 2007

Don't break the interface

Ian and Jonathon have been paired up on resolving a long standing omission from our system, aggregating inbound long messages.

Currently when a subscriber sends a long message to one of our SMS long numbers or short codes, each part of that message arrives in the inbox, or is pushed to our customers web service, individually. Aggregating the messages for multiple accounts while maintaining performance was a reasonably significant architectural problem to resolve.

We had looked at it before but were never quite happy with the solution. I finally committed the development time to do it as a result of the a case being raised by a very patient and reasonable customer. This time we were ready.

One of the constraints on any changes we make to our systems is preserving the interface to customers who use our API. In our initial plans we were going to potentially break it for some people.

Ian describes it in more detail in his post: Esendex Inbound Multipart Messaging Update. While we weren't strictly breaking the interface, if our customers had made reasonable assumptions about the length strings being passed it could break their application.

Not happy with this, the three of us sat down and we did come up with a solution that satisfied the internal architecture of our system while still supporting our existing customers.

We are going to provide an API option on an account to aggregate long messages. While we'll be storing messages in their aggregated form in our system. For those accounts with that option switched off, the API will re-split the messages before delivering them.

All existing accounts will have the option turned off by default and all new accounts will have the option turned on.

It makes the Options page a little more complicated but it keeps everyone working, which is what it's all about.

Friday, 17 August 2007

In an Extreme Programming world don't forget the value of traditional QA testing

When Extreme Programming is embraced, it is tempting to get carried away with the power of unit-testing.

If you're unit-testing everything across the whole build then surely everything should work. It is important to remember that unit-testing, when done properly, is just that. Testing units of code for compliance against a known set of outcomes. It doesn't necessarily test system wide interactions.

The development team here have recently spent a lot of time migrating our existing transaction management from the EnterpriseComponent, COM+ based, system to the TransactionScope framework provided in .Net 2.0.

Nicholas, Development Manager, as described the detail of the issues we've faced in his post: Transaction Scope - Hopes dashed. It has been horribly dissappointing, the performance profiling we did showed a big improvement, the deployment process was simplified. Everything seemed perfect.

The key to writing effective unit-tests are focusing on the unit itself and the logic in that unit and nothing else. This makes the test very quick to run, meaning you can run a full build, with tests in a short enough time to make you prepared to do it regularly.

But proper Quality Assurance testing picked up these issues. It wasn't until we ran these system wide test that the issues materialised. Generally this leads to further unit-test to ensure the issues don't materialise. Although in this case we decided to back-out this set of changes.

So unit-tests are an integral part of what we do, but good old grunt work testing is still the last, and most important, line of defence.

Friday, 10 August 2007

Create Mirror Copy of TiddlyWiki for Mobile Sync

In an attempt to resolve my synchronisation woes keeping a copy of my TiddlyWiki on my N95 I've added a new option to my TiddlyWiki file. This allows me to specify a path to create a mirror copy of the file. This points to my N95 synchonisation folder.

If I do work out how to enable editing on the phone I'll have to work out another approach but this does mean I can have a copy to read. Another benefit is that I can leave the backup copies accumulating in the folder or sub-folder and not have those copy across automatically as well.

I've created a new version of an empty TiddlyWiki and copied it to : ftp://ftp.esendex.com/adam/Tiddlywiki.AEB.v2.MirrorPath.html. The changes are summarised below.

Ln 571, added a new item to the config.options array:

txtMirrorPath: ""

Ln 791, added a new item to the config.optionsDesc array:

txtMirrorPath: "Path to mirror the file to"

Ln 831, added two new items to config.messages array:

mirrorSaved: "Mirror saved",
mirrorFailed: "Failed to save mirror file"

Ln 6152, added a new method saveMirror. Note this expects a complete path and not a relative folder as in the backup folder option.

function saveMirror(localPath,original)
{
 var mirrorPath = config.options.txtMirrorPath;
 var saveMirrorFile = mirrorPath != "";
 if (saveMirrorFile)
 {
 // add backslash if not present at end
     var backslashPos = mirrorPath.lastIndexOf("\\");
     var pathLastIndex = mirrorPath.length - 1;
     var backslashEnd = backslashPos == pathLastIndex;
     
     mirrorPath = mirrorPath + (backslashEnd ? "" : "\\");
     
 // copy to relevant locations
     var mirror = config.browser.isIE ? ieCopyFile(mirrorPath,localPath) : saveFile(mirrorPath,original);
     if(mirror)
      displayMessage(config.messages.mirrorSaved,"file://" + mirrorPath);
     else
      alert(config.messages.mirrorFailed);
 }
}

Ln 6133, added a call to the saveMirror method from the saveChanges method:

saveMirror(localPath,original);

Do let me know what you think.