Big data, NoSQL, and data science
Not all data fits neatly into tables, and some datasets are too large for one machine. New tools and fields have grown to store and learn from data at enormous scale.
Big data
Big data refers to datasets too large, fast, or varied for traditional tools. It is often described by volume (amount), velocity (speed of arrival), and variety (different forms, such as text, images, sensor readings, and clicks). Examples include social media activity, scientific instruments, and logs from millions of devices.
Distributed storage and processing
Big data is spread across many machines. Distributed file systems split files into blocks stored, with copies, on many servers. Processing frameworks, such as MapReduce, popularised by Google in the 2000s, and later Apache Spark, divide computations into tasks that run in parallel near the data, then combine the results.
NoSQL databases
NoSQL databases relax the relational model for flexibility or scale. Key-value stores, such as Redis, map keys to values very quickly. Document databases, such as MongoDB, store JSON-like documents with flexible structure. Wide-column stores, such as Cassandra, handle vast write volumes. Graph databases, such as Neo4j, store nodes and relationships for highly connected data.
Trade-offs in distributed data
The CAP theorem says that when a network partition splits a distributed system, it must choose between consistency (every read sees the latest write) and availability (every request gets a response). Many large NoSQL systems favour availability and accept eventual consistency, where copies agree after a short delay; banks usually favour strong consistency.
Data science
Data science combines statistics, programming, and domain knowledge to extract insight from data. A typical workflow is: ask a question, collect data, clean it (often the longest step), explore and visualise it, build models, evaluate them, and communicate results. Python and R are the most common languages, with libraries such as pandas for data handling.
Machine learning
Machine learning lets computers learn patterns from data rather than following hand-written rules. In supervised learning, models learn from labelled examples, such as emails marked spam or not spam. In unsupervised learning, they find structure in unlabelled data, such as grouping customers by behaviour. Deep learning uses many-layered neural networks and powers image recognition and language models.
Data quality and responsibility
Results are only as good as the data: 'garbage in, garbage out'. Missing values, errors, and unrepresentative samples produce misleading conclusions, and biased training data can create biased models. Responsible data work also respects privacy and laws such as the EU's General Data Protection Regulation (GDPR), collecting only what is needed and protecting it.
Worked example
A shop wants to predict which customers will stop buying. Data scientists combine purchase histories, website visits, and support tickets; clean duplicates and fix missing dates; and find that customers who stop visiting for 60 days often leave. They train a supervised model on past customers labelled 'stayed' or 'left', test it on held-back data, and send at-risk customers offers, then measure whether the offers actually help.
Key takeaways
Big data is characterised by volume, velocity, and variety. Distributed file systems and frameworks such as MapReduce and Spark process data in parallel across machines. NoSQL includes key-value, document, wide-column, and graph databases. The CAP theorem forces a choice between consistency and availability during partitions. Data science runs from question to communication, with cleaning often the longest step. Machine learning can be supervised or unsupervised. Data quality, bias, and privacy laws such as GDPR matter.
Flashcard
Data too large, fast, or varied for traditional tools
Check your understanding
A model is trained on emails already labelled 'spam' or 'not spam'. What kind of learning is this?