Information-Driven Business
Robert Hillard

It’s time for a new definition of big data
by Robert Hillard

Two words seemingly on every technologist’s lips are “big data”.  The Wikipedia definition for big data is: “In information technology, big data consists of datasets that grow so large that they become awkward to work with using on-hand database management tools”.  This approach to describing the term constrains the discussion of big data to scale and fails to realise the key difference between regular data and big data.  The blog posts and books which cover the topic seem to converge on the same approach to defining big data and describe the challenges with extracting value from this resource in terms of its size.

Big data can really be very small and not all large datasets are big!  It’s time to find a new definition for big data.

Big data that is very small

Modern machines such as cars, trains, power stations and planes all have increasing numbers of sensors constantly collecting masses of data.  It is common to talk of having thousands or even hundreds of thousands of sensors all collecting information about the performance and activities of a machine.

Imagine a plane on a regular one hour flight with a hundred thousand sensors covering everything from the speed of air over every part of the airframe through to the amount of carbon dioxide in each section of the cabin.  Each sensor is effectively an independent device with its own physical characteristics.  The real interest is usually in combinations of sensor readings (such as carbon dioxide combined with cabin temperature and the speed of air combined with air pressure).  With so many sensors the combinations are incredibly complex and vary with the error tolerance and characteristics of individual devices.

The data streaming from a hundred thousand sensors on an aircraft is big data.  However the size of the dataset is not as large as might be expected.  Even a hundred thousand sensors, each producing an eight byte reading every second would produce less than 3GB of data in an hour of flying (100,000 sensors x 60 minutes x 60 seconds x 8 bytes).  This amount of data would fit comfortably on a modest memory stick!

Large datasets that aren’t big

We are increasingly seeing systems that generate very large quantities of very simple data.  For instance, media streaming is generating very large volumes with increasing amounts of structured metadata.  Similarly, telecommunications companies have to track vast volumes of calls and internet connections.

Even if these two activities are combined, and petabytes of data is produced, the content is extremely structured.  As search engines, such as Google, and relational databases have shown, datasets can be parsed extremely quickly if the content is well structured.  Even though this data is large, it isn’t “big” in the same way as the data coming from the machine sensors in the earlier example.

Defining big data

If size isn’t what matters then what makes big data big?  The answer is in the number of independent data sources, each with the potential to interact.  Big data doesn’t lend itself well to being tamed by standard data management techniques simply because of its inconsistent and unpredictable combinations.

Another attribute of big data is its tendency to be hard to delete making privacy a common concern.  Imagine trying to purge all of the data associated with an individual car driver from toll road data.  The sensors counting the number of cars would no longer balance with the individual billing records which, in turn, wouldn’t match payments received by the company.

Perhaps a good definition of big data is to describe “big” in terms of the number of useful permutations of sources making useful querying difficult (like the sensors in an aircraft) and complex interrelationships making purging difficult (as in the toll road example).

Big then refers to big complexity rather than big volume.  Of course, valuable and complex datasets of this sort naturally tend to grow rapidly and so big data quickly becomes truly massive.

comments powered by Disqus

blogThe Information-Driven Business blog is published monthly:

   2019   2018   2017   2016   2015
   2014   2013   2012   2011   2010

Also featured from the Information-Driven Business blog:

Maturing the information economy
Today it costs a fraction of a fraction of a cent per megabyte to store data, but it is less than 30 years since the cost of data storage dropped below one US dollar per megabyte. It seems that one … Continue reading

Common sense has never been more important
The administration of day-to-day life seems like it has never been more complicated. We all seem to be constantly navigating new services, products, registrations, cards, identification and so much more. New requirements are being layered over the top of basic … Continue reading

Opportunities beyond startups
Is it just me or has the world gone mad for startups and writing software? Don’t get me wrong, I am a big fan of startups and all that they bring to the economy. However, if you read the business … Continue reading

Email works too well
Everyone who regularly feels overwhelmed by their email would agree that there is a problem.  The hundreds of articles about the issue typically make the same assumption and are wrong. Writer after writer bemoans email as inefficient and an obstacle … Continue reading

The Internet was a mistake, now let’s fix it
Each generation over the last century has seen new technologies that become so embedded in their lives that its absence would be unimaginable. Early in the 20th century it was radio, which quickly become the entertainment of choice, then television, … Continue reading

© 2010-2019 Robert Hillard