Every where you look at the moment you see and hear about ‘Big Data’ and how its the next biggest thing to transform society and businesses. Initially I just took the term to mean just more data from more varied sources such as social media and doing the same sort of thing that is normally done with data. Of course anybody ahead of me on the curve, which I suspect is most people, will know that is really underestimating the value and worth of Big Data. However, the term has become so omnipresent that I thought it about time to start looking into what it actually is and how it is (allegedly) going to transform everything.
The term Big Data is not the best term and possibly one reason for my complacency up till now. There has always been large amounts of data around such as that from loyalty card schemes and the like, so why the new term Big Data? Firstly there is so much more of it around, but not only that but its velocity, variation and is inherently unstructured.
Its these very qualities of Big Data that means a different method of handling and processing it is required. Traditionally, relational databases were used to store structured data but the problem here is that they don’t scale very well and are expensive. Enter Hadoop and MapReduce, NoSQL and MongoDB. But I am getting ahead of myself; lets look first at some benefits of Big Data.
It has been said that about 90% of all data has been generated in just the last few years. So where is all this new data coming from? You only have to look at the explosive growth of social media, emails, texts and online shopping as just a few sources of all this data. Another source of data set to grow exponentially apparently is that from the ‘Internet of Things’ or IoT, which is something I will be covering in future posts. Though it has been claimed by some that IoT is just a ploy by some manufacturers to generate additional markets! Add to this the growing number of publicly available data sets from the open data project and similar schemes. Well, I think you are getting the picture by now. Some readers may be thinking ‘Big Brother’, and yes that is another very real possibility we have been reminded of recently courtesy of Ed Snowden. Why else would the NSA require the huge yottabyte data centre codenamed Bumblehive in Utah?
According to one report, the companies who have already adopted Big Data are beginning to outperform their competitors by a country mile. From insurance companies determining risk profiles and driving behaviour to fraud reduction by financial institutions to name just a few.
Big data is not just about historic business intelligence but also the addition of real-time data and mashing it all together to produce data sets that can be analysed to make manufacturing, production and marketing more effective. However, for those wishing to jump on the bandwagon its not just a question of passing it off to the silo that is the IT department, it requires new and multi-discipline skill sets and needs to be embedded into the organisation as a new paradigm for doing business.
I hope you will join me in my upcoming posts where I hope to learn all about the technology and applications behind Big Data and review some relevant books along the way.