Dipping into Kubernetes & Containerisation
WhereIsMyTransport is a big data platform for sustainable mobility in emerging markets. From Africa to India, Latin America to Southeast Asia, we’re working to make formal and informal public transport reliable, predictable, safe, inclusive and accessible; so that everyone in the emerging world can get where they want to go.
We’re doing this by mapping all of the public transport networks in emerging markets and making the data available through our integrated mobility data platform. Governments and service providers use our data and technology to build sustainable urban mobility plans, and millions of people riding public transport use it to make smarter transport choices.
We currently have data from 36 cities in our platform, and we’re expecting to get close to 100 by the end of the year. That kind of expansion requires architecture robust enough to store the largest source of the world’s most complex public transport data for emerging markets, and to share that data with services that need it.
Meet Dave
OOur co-founder and software architect, Dave New, is overseeing the project. He recently sat us down to explain some of the many benefits of adopting Kubernetes, focusing on how it helps us optimise costs as we scale our data and technology.
It’s a whale of a project
Our platform is made up of many systems, all working together in concert to support the services we provide. Each of these systems exists in its own environment. High-performance environments make for healthy, high-performance systems, and we spend a lot of time and money looking after both.
In Dave’s analogy, we imagine that each of the systems is a whale, and its environment is an aquarium; a fish tank.
We breed the whales ourselves, but we pay service providers — like Microsoft Azure — to provide the fish tanks.
Supporting Scale
Every tank must be big enough to support a happy, healthy whale. If the tank is too small, the whale becomes unhappy; and stressed. In the software world, this translates into system crashes, performance issues and maybe even downtime. Which is bad.
You can solve this, of course, by buying bigger tanks. That’s expensive, and it’s not always an efficient use of space. Tanks come in predetermined sizes, so more often than not, you would end up with bigger tanks than you really needed — and a load of unused (and expensive) extra space.
Whales also grow in size — sometimes quite quickly, and not always predictably — so you’d want to budget for that… which means even bigger tanks today, in anticipation of probable growth tomorrow.
All this makes it difficult to optimise for today and tomorrow without spending a lot of cash.
And that’s one of our big challenges. As we collect and upload data from 100+ cities onto our platform, we’re going to be looking after more and more whales, all growing in size. And, like any business that’s scaling up, we have to watch every penny we spend.
So we need a better way to host and deliver our services and to keep these big fellas happy and healthy.
Enter Kubernetes
Kubernetes is the answer for us. It allows us to host more than one whale in a single, scalable tank. If more of our whales can share the same aquarium, it means we can optimise on the cost of these resources, particularly as we grow. It’s also easy to add new tanks, and Kubernetes takes care of moving whales between different tanks, to optimise performance.
This all adds up to a neat economy of scale for us. We cut the overhead of maintaining many small tanks — all different shapes and sizes — and of managing upgrades tank by tank. Less aquarium management means less overall hassle, and it also reduces the number of things that can go wrong. Good news for us, great news for the millions of people who rely on our services.
We’re most of the way through our transition to Kubernetes. Containerisation is a new concept for the team, and we’re learning a lot during the transition. Some lessons are harder to learn than others! But we’re already seeing better utilisation of our infrastructure, and the real benefits in cost and scale are still to come.
More whales, fewer tanks, might seem like an obvious solution, but actually, Kubernetes is the first solution that isn’t an expensive headache to implement for the systems we run.
No whales were harmed in the making of this post.
TA big thank you to Docker for inspiring us to use whales to explain Kubernetes & Containerisation.
