Big Data Demystified: What It Is and Why You Should Care

You may have heard the term “big data” floating around especially when it comes to companies looking for skills around big data. Here is my short guide on what the heck “big data” actually is, is it a hype word? Is there a new word for something that’s been the same for a long time?? Read on… The word data is super generic, the Merriam Webster dictionary quotes the definition of data as: 

“Factual information (such as measurements or statistics) used as a basis for reasoning, discussion, or calculation” 

Merriam Webster Dictionary

Ooh, sexy right? I guess if you’re someone who loves a good stat or measurement, or fact, then yes. Anyway – data we know what data means, but why do I need to know what BIG data is? Businesses collect data on their customers to spot patterns, this is to better help them grow in their area, this may be from how they sell products to you or to understand profiles on unknown such as in the medical field – data is the fundamental when it comes to growing. However not all companies have put data as a priority in their business, and generally businesses suffer because of it. Two points from this:

1. Data is so important in business 
2. Knowing this stuff(DATA!), can put you ahead of other candidates for jobs when you have a firm understanding of what problems you’re probably walking into. 

So what is BIG data? 

At a high level, I mentioned above that businesses collect data on customers. This could be anything from profiling a customer (age, gender, interests, likes, environment, purchase histories, etc) can you imagine how much data is collected on one customer here? Imagine how many customers companies with large customer bases have… that is some BIG data right there. 

Diving more deeply into data, and looking at how it is structured when stored, imagine a company collects date of birth of customers, for a few reasons, one is to collect the ages of it’s customers. by taking the date of birth and doing a calculation against that day’s date, it can generate an age, and also use date of birth for something else. It may store the date of birth in a raw format (as is, as numbers without any separation (like DDMMYYYY)) then there may be some work to make it a bit cleaner and optimise performance if someone were to query the data in the company, well there you already have multiple versions of this data and then imagine that for 100’s or 1000’s of customers. 

Why use big data? 

The insights gained from big data analytics can be used for various purposes, such as improving business decision-making, detecting fraud, predicting trends, and developing new products and services.

What differentiates big data from… data? 

Big data refers to extremely large and complex data sets that are beyond the ability of traditional data processing software to manage and analyse. The term “big” refers not only to the size of the data, but also to its complexity, diversity, and rate of growth. 

To make this more simple, we can characterise Big data by the three Vs:

Volume: Big data refers to massive amounts of data that can range from terabytes to petabytes or even exabytes.

Variety: Big data is characterised by the diversity of data types and formats, which can include structured, semi-structured, and unstructured data from various sources such as social media, sensors, and log files.

Velocity: Big data is generated and processed at a high velocity, often in real-time or near real-time. Basically how quick to process the data! – have you tried to load a browser with 100 tabs – that could be a bit like the big data, so keeping the processing time down, keeps costs down to process all that data.

 

What challenges does this present? 

The challenge of big data is not only storing and managing the data, but also processing and analysing it to extract valuable insights and information. Big data analytics involves the use of advanced technologies and techniques to extract meaning and insights from large and complex datasets. 

What next? 

Make sure you sign up to my email list to get alerts on the latest blog posts. If you’ve enjoyed this blog, why not let me know? Feel free to reach out to me on Instagram! @techy.rey