Building super fast, super portable mobile maps - Part Three - Making it super fast!

So, my maps are scrolling around my mobile screen really nicely. I'm jealous of how fast GMM is, and I can't see how they've got it so fast.

Jaak Leineste again had the solution, in this post

" We have solved it with the Mgmaps lib SDK and CloudMade tiles this way, that we set up own "tile streaming" server to AWS EC2, so it has superfast connection to CM tile server, and I am doing the tile proxying there just like you describe. I must admit that I got also some inspiration from the google's mobile maps app :). You can compare different map reading options, direct 64x64 tiles or the multi-tile requests (we call it "streamed") with on-line web runner demo at http://www.nutiteq.com/libsamples.html#BasicMapperapplication . Start the app from web (it starts microemulator automatically), you see performance with default map, which is CM 64-pixel individual tiles. Select Menu > Change map > select CloudMade streaming (via Nutiteq) and see speed then. Difference is quite big, to say the least. Not worse than with google maps mobile."

So this is the secret to super-fast mapping - you have a webservice which fetches tiles for you and concatenates them into a single download. You then pull them out of the stream. The reason the old method is slow isn't because of the speed of my connection on my mobile, its the latency in setting up the http request. If I've got fewer requests, then I've got a faster map.

My problem was that the MoSync image downloader doesn't provide me access to the stream. It just gives me a formatted PNG image I can display. I needed to write my own downloader.

MoSync does provide access to their connection object however. With this, I can have direct access to the download screen. So with the help of Niklas and Fredrik at Mobile Sorcery, I wrote my own downloader which would retrieve tiles out of a stream.

I wrote my own streaming service in C#. This takes my API key and the details of the tiles I want from CloudMade. It fetches them in order from the CloudMade server, and returns them back to the requestor, flushing the response stream after each tile so that the client application isn't waiting needlessly.

This has made a massive impact on my mapping component. Previously it would take 30-60 seconds to get all 36 tiles. Now I get all the tiles I need on the screen in < 10 seconds, and all 36 tiles in < 20 seconds. Personally, I think that my component is at least as fast as GMM.

Building super fast, super portable mobile maps - Part Two - Getting Maps onto the Screen

So, I've got a client who wants mobile maps, and I've got a supplier of map tiles. I've signed up as a developer on CloudMade, got my API key and I can see tiles. Now I've got to get them on screen.

About a 40 years ago (ok, about 25 years ago maybe) I was working out how to get my Acorn Electron to play Chess. I was just hacking around more or less directionlessly, and I fell into a classic programming trap.

My thought process was: "A chessboard. That's an 8x8 grid. So I'll represent the board in memory with a two-dimentional array. 10 DIM BOARD[8,8]". This seems obvious, but it is not the easiest way to implement a grid where you want things to move from one place to another place, and it isn't too quick in implementation. For example, to calculate valid knight's moves becomes fairly complex. If I've got a knight at [2,0] then I need to try permuations of where it can go in both dimensions. If I created a board as a single dimension array (10 DIM BOARD[64]), then I know that valid moves are:

currentLocation + 6, currentLocation + 10, currentLocation + 15, currentLocation + 17 currentLocation - 6, currentLocation -10, currentLocation - 15, currentLocation - 17

provided that the result is between 0 and 64.

Similar decisions are needed when you create a map grid on a mobile phone. So, again I started with an assumption of having a 240x320 screen. If I imagined a grid of 6x6 squares for the map tiles, then that will give me a total grid size of 384x384 - enough to cover the whole screen, plus a margin all around so that when the user scrolls, the tiles are ready. If I create this grid as a single object array[36] then it makes life really easy when it comes to scrolling. More on this later.

So I created a class to represent a MapTile, a marshaller to handle the downloads, and a Widget to put the map on screen. The widget converted the location from the GPS (or location API) into a CloudMade tile reference, and then took this as a basis for calculating the references for all the other tiles.

Each MapTile is responsible for formatting a URL which will get the image from the CloudMade server. It requests a download from the marshaller, and in turn gets its image.

The widget then puts each image on screen as they are downloaded, building the map. It also handles the scrolling, where the smart decision about an array MapTile[36] pay dividends.

To put the map on screen, I need to know which tile is going into the top-left corner. I call this the 'origin tile'. When my map starts this is the tile at MapTileArray[0]. The tile with the actual location I want to see is MapTileArray[14], which is about the middle of the screen. I start by displaying MapTileArray[originTile] top left, and then rip through the array in order, adding 1 MOD 36 to get to the next tile. This means when I get to MapTileArray[35] and want the next tile, I know that the next tile is MapTileArray[0].

I also wanted it to prioritise which tiles to get first. I don't want it to start with the top left tile and work sequentially though. I wanted it to start with the tile which shows the relevant location and spiral out. Initially, I worked on an overly complex algorithm which will calculate the order that the tiles should be downloaded in at runtime. In the end, I got some common-sense together, and just set up the order in code. I add each of these values to the value of the origin tile, MOD 36, and that is the order of the tiles that I want.

If I'd gone with a two dimensional array, then I've got an array I want to reflect on screen. This means that a scrolling operation doesn't mean adding to subtracting either 1 or 6 (depending on the direction of the scroll), but I've got to switch all the tiles into different locations into the array to maintain its reflection of the screen. Massive pain.

So I can scroll my map around nicely, my array is working quickly and easily. Its a lot slower than Google Maps though, and I start to wonder why.

Part Three >>

Building super fast, super portable mobile maps - Part One

I've spend the last couple of weeks building mapping into one of my projects. It is something I've resisted in the past as being too complicated, too difficult and not offering a lot of value. However, my client wants a location based messaging solution, so maps we're a must have.

My client (who shall remain nameless for now) said that it was easy. His previous partners had done it really quickly; just mash-up Google Maps. The problem is that this is a very web-oriented view of development. Whilst GM are easy to mash-up in a web environment, they aren't licenced for mobile application development. Their API is also Javascript-based, so I wasn't going to be able to use that either.

So the next stop is just to hack it. I found that many people have done this in the past. You just need to know how Google Maps for Mobile (GMM) converts co-ordinates into map tiles, and you can build URLs to get the correct tiles. I also discovered that everyone who had done this had quickly received 'cease-and-desist' letters from Google's lawyers.

So what are the alternatives? Yahoo! Maps are as rigidly licenced as Googles. Microsoft are much more relaxed about accessing their geodata, but they only make tiles for the web.

The issue here is the size of the tile. Let's assume for the sake of argument that the normal size of a mobile screen is 240x320 pixels. The tiles used on the websites are 256x256 pixels, which means that one tile will cover almost the whole screen. If you've only got a 128x128 screen, then you've got something which is way to big, giving the developer the additional headache of working out which bit of this tile needs to be on screen. The main problem here is that this is one download which means a big delay between hitting the button on the phone and seeing the map. GMM has mobile-friendly map tiles sized at 64x64 pixels. This means that I can convert my location to a single 64x64 tile, and centre my screen around this.

So a quick bit of research later brought me to the OpenStreetMaps initiative. This is crowd-sourced geo-data. You go out with your GPS and send data about all the streets in your neighbourhood. This becomes 'open sourced' data. Digging a bit more lead me to CloudMade, the extension of OSM to convert their raw data into useable map tiles.

CloudMade have fantastic support for mobile. Not only do they do their tiles in 256x256 and 64x64 pixel tiles, but they also do their mobile tiles with high-contrast colour schemes. Now they are extending this, so anyone can create custom cartography with their style editor.

Better news seemed to be that Jaak Laineste has developed a J2ME component from his compay Nutiteq. This wraps up all the functionality you need to make your own GMM-clone with maps from CloudMade. This would be a massive short-cut, except that I'm not writing in J2ME; I developing in MoSync.

<digression>

The number one problem with mobile development is that every damn phone is a little bit different. The purpose of Java is that you write your code once, and it will run in an identical fashion on a different platform. The problem with this is that it has never quite been true, and for the Java you've got on your phone, this problem is magnified by a thousand times.

The problem is that you've got one body specifiying the language, and dozens of bodies implementing it. Just as no two people will interpret a book in the same way, no two implementers make J2ME work in exactly the same way.

The upshot of this is that even when you've got your J2ME application working on your phone, there is no guarantee that it will run on anyone else's. Or a Windows Mobile device, or in an S60 environment.


The guys at Mobile Sorcery have taken this pain away. They've written their own API which they've tested on the handsets. The difference with this approach is that it is one body who is managing implementation, massively improving compatibility. You write (or port) your code in C/C++ against their API, and it will build your application for hundreds of phones. Each of your target handsets gets a build specifically for it. On J2ME phones, it builds a J2ME application. On Windows Mobile devices, it builds a Windows application. On S60 devices it builds a Symbian application.

</digression>

Still, even without being able to use Nutiteq's mapping component, I've got a supplier of map tiles.

Part Two >>>

The Three Roadblocks to Mobile Web

I seem to be giving this talk a lot lately, so I thought I'd pull my finger out and blog something.

Many handsets have had web connections (and actual http/html connections, not just wap) for many years. On some devices its even quite good. But it has never made the penetration into normob psyche that many people predicted. I'm often asked why this is, and why I continue to evangelise on-device portals and applications over mobile web given in availablity of all-you-can-eat data plans.

To illustrate my point, I distilled my thoughts down into what I call 'The Three Roadblocks to Mobile Web'.

So you've built a brilliant mobile web site. Perhaps you've used one of the automated tools to rebuild your main site and represent it to a mobile user. The problem is, you're not getting any hits. What is the problem? Does your content suck? You are getting loads of traffic through traditional routes, but not through mobile. How can you get more people to your site?


1. Having the idea

The number 1 problem with mobile websites, is to get people to have the idea to use their phone to do some web surfer at a convenient moment. I used to use the example of someone waiting on the platform for their train, but Paul Golding recently pointed out to me that the peak time for mobile web use is between 23:00 and 00:00 - when people are in bed. Anyway, you can imagine your ideal user who is idling for a few minutes - now is the perfect time for them to visit your website. The problem here is that even when people are in this situation, even when they're playing with their phone, you've still got to get them to press that Internet button.

There are a lot of people who are scared of this button. There persists an idea that connecting to the web is expensive and unpredictable. Even my wife, who I consider do be reasonably technical, occasionally asks me how long I've been reading that article, like it billed on connection time.


Having an on-device application puts your content in view. If your content has been downloaded and cached to be read offline, then people are much more confortable about starting the app. The fear that you don't know how much it is going to cost to click the next link disappears, and your content icon is one of maybe a dozen application icons on most phones, and not one of a million billion sites.

2. Knowing you exist

So, even if you've got someone in bed doing a bit of late night surfing. How do you get them to browse to your site? Mobile traffic isn't driven in the same way as desktop traffic. You can have all the SEO and tagging and cool tricks in the world, but the fact remains that typing a couple of keywords into that Google box you've got at the top of your browser is very quick and easy, and typing even one keyword into the Google box on your phone is more trouble than you can be bothered with, even if your really interested in the result.

3. Getting to your site

Traffic is driven from operator portals. Your late night surfer is going to see a couple of things on their home screen that catch their eye and click through on those. They're unlikely to go off that. If they can't be bother to search Google from their phone, they're definately not going to type your URL in.

So you spend some money, and you get a keyword on a shortcode. You advertise it, and the auto-responder sends your URL back. And some people even save it. In the list of bookmarks in their browser. Out of sight, out of mind.

So this is why I back on-device applications.

1. It is visible - you've put your icon and your brand literally into the pockets of your readers/customers
2. It is memorable - you're application is now in direct competition for eye-time with your user's browser. You're way up the decision making tree now.
3. It is comfortable. I build ODPs to download and cache data in a controlled way. This isn't necessarily any less data, but it is more predictable for the user. They feel in control, and with off-line browsing there is no fear in pressing the next link.

And if you're thinking of building an app store, then why are you thinking of building it as a website? Put your icon on your customers phone.

I buy a lot of games from GameMobile, but I wish that they'd build their own app so I can browse the latest games comfortably from my phone, rather than on line. I promise you that I for one will be hitting that 'Buy Now' button far more often.

What I've got against NHibernate

Apart from being another example of too much configuration in XML? Well Billy McCafferty has summed it up for me in his article on CodeProject

Your database should be an "implementation detail" that is defined to support your domain model, not the other way around.

No it shouldn't. This is backwards for all but the largest businesses with lots of DBAs and data warehouses. This is a very arrogant developer argument to make.

It is more important to the business to be able to quickly and accuratly report about the business than to sell one more widget. Getting timely and accurate information about the widgets you've sold is more important. I know of lots of businesses who sold lots of their product and still went bust, owing to poor finanical management. Every business has to be able to manage their cash-flow at least as well as they can sell, and the database should represent this. Programmers can make the program more flexible to fit in with the best design for the business, but database design should be driven by business requirements, not programmers' whimsey.

In many, many cases both of these requirements will be in alignment, but not always. Some of your objects may need to populate and be populated from multiple tables, and whilst NHibernate can support this, it does drive developers to generate a 1-1 database model matching their entities.

And another thing, Billy doesn't hint at this, but says it explictly.

it is commonly accepted by many that ORM technologies, when used correctly, are, in fact, a silver bullet for software development

No they're not - don't be stupid. ORM technologies are a useful shortcut for developers to do less boring code, but they are not a silver bullet. They do not replace data abstraction layers, mearly form part of it. ORM is about being able to maintain the state of your objects in a database. Databases do more than just store developer's objects - how arrogant is this man?

Databases and DAL often perform complex data manipulation required by the business. Sometimes you need a process which will derive data from inputs, and you need a manageable, maintainable design pattern to implement these processes with data decorators and transformation objects. Whether you choose to put this code in NHibernate or elsewhere, then it still needs to be done. Using NHibernate doesn't remove business requirements.

Dependency Injection Frameworks

Call me crazy, but I don't get why the world has gone mad, and experienced developers are in love with DI frameworks.

Don't get me wrong - I get IoC. I can see how it helps with flexibility even if I'm not completely convinced that it reduces coupling. You're, at best, no better off in terms of coupling, but objects can be used in new abstracted ways.

I even get understand how dependency injection could potentially help this, adding another layer of abstraction on how your objects relate to each other.

What gives me the fear is that you're putting your application configuration into an XML document. That's an XML document everyone, which is going to control how your objects relate to each other; which types to create; which objects should be cast with which interfaces. In an XML document.

A document that maybe dozens of developers will be working on. A document that is very, very unlikely to contain any comments. A document which is not validated at design time, or build time, only at run-time. A document which in twelve months will be unmaintainable garbage. A document that contractors will dick with for a few weeks and then leave. Your entire application in the hands of one XML document. No.

Lots of things DI framework providers write really scare me, and I don't scare easily. Here's something from Winter4Net

Fast - handles more than 20mb configurations

Aarrrghhh! I'm a big tough man, but even I tremble when I think that my enterprise-scale application maybe in the hands of a 20mb XML document that no-one will be able to follow.

And actually, its not just mentally-large config documents. Here's what the ubiquitous Nate has to say about Ninject.

To define a conditional binding, the two fluent interfaces (binding and condition) work together. Here's an example:
Bind().To();
Bind().To().Only(When.Context.Target.HasAttribute());

What? So now I'm putting business logic into configuration? I admit that if I had to use a DI framework, then I'm attracted to Ninject if only because of the whole 'it's not an XML document'

How can we take this idea further? I know! What I'm going to invent is a really flexible configuration system which I'll process and turn into commands my CPU understands, and I'll call it F*CKING CODE!