The rise and rise of Big-Data

By Sean Fanning | Oxford Knight


Think for a moment back to when you were 18 years old. How much data did you put on the web? A few paragraphs on Myspace, the usual emails, maybe an E-Bay ad or even a blog like this. Now think of today’s 18 year old – by and large using Twitter, Facebook, Youtube etc on a daily basis, and in doing so putting huge amounts of content onto servers globally. This exponential rise in content has given rise to the phenomenon of Big-Data. Put simply, according to IBM, we now create 2.5 quintillion bytes of data/day.

What is Big-Data?
Global companies, institutions and each of us produce ever increasing amounts of data from online purchases to personal videos to climate sensors. Not only does this data need to be stored somewhere, but more importantly this data holds true value as enormous data sets in which patterns emerge and valuable, intelligent predictions can be made.

In the truly globalised digital age we inhabit data takes many forms; structured and unstructured, temporal, geospatial, disparate and unconnected. Big Data technologies not only store this data but critically allow companies to wrangle data quickly and efficiently in ways that were traditionally out of reach. Quantitative analysis has become much more accessible.

Big Data Technologies
Far and away the most famous and most-used Big Data technology is Apache Hadoop which is an open-source framework that supports data-intensive distributed applications. Hadoop is run over a large cluster of computation-independent computers and allows applications to work with petabytes of data.

Essentially the framework implements a computational paradigm named Map/Reduce, where the application is divided into many small fragments of work, each of which may be executed or re-executed on any node or in the cluster. It is derived from Google’s MapReduce and contains Hadoop Kernel / MapReduce and HDFS as well as Hive, HBase, Zookeeper and so on.

Yahoo! inc were early pioneers but unsurprisingly Facebook now claim to have the largest Hadoop cluster in the world with 100PB of storage.

Practical Applications

Content Delivery – Firms that are now commonplace such as and Netflix have a lot to thank Big Data pioneers for as it has been the key technology underpinning their growth and total accessibility. To be able to deliver huge amounts of content in real-time to distributed users at their timing request is an engineering challenge that could only be overcome by Big Data technologies. Netflix stores over 1 PB of data on Amazon’s hosting cluster in North America.

Analytics – The rise of opensource Big-Data technologies and the reducing cost of running complex computations has sprouted a rise in firms that specialise in analysing both industries, institutions and also individuals. So whether it’s Klout who analyse the impact of individuals through Social Media and monetise it; or DataSift who extract insight from billions of public data points on Twitter, there is a company analysing your public data somewhere!

Internal Analytics – The biggest companies can streamline and massively improve efficiency within their sales and organisational structure by employing quantitative analysis on a huge scale. Examples include power companies predicting power usage through analysing over 400 million power meters.

Big Data Developers

There has been a steady increase in the amount of developers who are specialising in Big-Data technologies and there is very little to suggest that this won’t increase. As the paradigms of distributed computing and Big Data take much deeper root in Academia engineers will be exposed at an earlier stage and in general the standard of engineering will improve. Moreover specific to the UK is the Valley shift to London whereby Silicon Valley innovation is mirrored and improved upon by the next generation of entrepreneurs. Indeed many of the Silicon Roundabout firms harness Big Data or specialise wholly in it meaning there are many jobs and good prospects in London for those with particular interest in data wrangling.


So then, is Big Data a modern version of the DotCom bubble? While the proliferation of firms pertaining to be doing something original with data reduces and the market congests, the fundamental driving force behind it shows no signs of stopping. Data is and will continue to be essential in this modern digital age and with millions more accessing the web and creating data each day, every indication would suggest those developers that specialise in Big Data have a bright future.

Here too the market will become congested and contractors will become prevalent as the big banks and financial firms finally cotton on. Given the way technology in general is developing, there is nothing to say that we shouldn’t expect new computational paradigms to develop and flourish over time. But for now – Big Data is Big Business and it most likely will be going forward too.

Want to Find out More?

So whether you are a Big-Data specialist or an accomplished engineer looking to stay ahead of the curve and enter this exciting and challenging market, I am sure that Oxford Knight will be able to give you excellent advice and guidance. If you are a potential candidate or a new firm with a need to find relevant and specialised candidate don’t hesitate to get in touch.

By Sean Fanning




Oxford Knight is a technical recruitment agency. None of our consultants have written a line of code… yet. We apologise if this article doesn’t keep some purist happy, but we’re trying to build a new generation of technical recruitment agencies…. We listen, participate, and deliver.

More information

Get in touch using the contact form below or give us a ring at 020 3137 9570. We'd love to speak with you about what we can offer.