Big Data refers to datasets whose size, speed, or complexity make them difficult to collect, store, process, or analyze with traditional tools. The challenge is primarily computational, but it has strong implications for statistics and machine learning.
Big Data is commonly summarized by the 3 Vs:
Sometimes other Vs are mentioned:
(not discussed in this course).
# Years
years <- 2010:2029
# Data volume in Zettabytes
datasphere_zb <- c(
2.0, 5.0, 6.5, 9.0, 12.5, 15.5, 18.0, 26.0, 33.0, 41.0,
64.2, 79.0, 97.0, 120.0, 173.4, 181.0, 221.0, 291.0, 394.0, 527.5
)
# Colors: historical (<=2024) blue, projections (>2024) orange
colors <- ifelse(years <= 2024, "blue", "orange")# Plotly bar chart
library(plotly)
plot_ly(
x = ~years,
y = ~datasphere_zb,
type = 'bar',
marker = list(color = colors)
) %>%
layout(
title = "",
xaxis = list(title = "Year"),
yaxis = list(title = "Data Volume (Zettabytes)"),
template = "plotly_white",
annotations = list(
x = 2010,
y = max(datasphere_zb) * 0.95,
text = "",
showarrow = FALSE,
xanchor = "left",
yanchor = "top",
font = list(size = 10, color = "gray")
)
)
The global population is increasingly online, which means that every minute, a huge amount of data is generated on the internet.
The following figure illustrates the amount of data generated every minute of every day:
Big Data is often collected for operational purposes, not for statistical inference.
Data is often private, but some websites allow public access to large datasets in the perspective of Open Data.
In this course, we will often assume that each of the \(n\) observations can be represented as a vector
\[
X_i \in \mathbb{R}^p.
\] In other words, a fixed number of numerical variables is measured for each individual.
However, many modern datasets do not fit into this framework. Instead of vectors, observations may be:
Such data are referred to as complex data.
Most classical methods covered in this course (PCA, LDA, CART, Random Forests) assume that we observe independent discrete random vectors in a Euclidean space.
Complex data invalidate this assumption. We need to understand their structure before applying any statistical or machine learning method.
This leads to new representations, metrics and notions of variability that are beyond the scope of this course.
In functional data analysis (FDA), each observation is a function: \[ f_i(t), \quad t \in \mathcal{T}, \] where \(\mathcal{T}\) is typically time, space, or wavelength.
Examples:
Each observation is not a vector, but an entire curve.
Key challenges:
Typical tools:
Functional data emerged as a consequence of the rising number of IoT devices monitoring variables at a very high frequency, that are better analyzed as continuous curves rather than discrete numerical values.
In distributional data analysis (DDA), each observation is a probability distribution: \[ \pi_i (x), \quad x \in \mathcal X \] (taking positive values and integrating to 1) rather than a single realization.
Examples:
Key challenges:
Typical tools:
In graph data, each observation is a network: \[ G_i = (V_i, E_i), \] with nodes \(V_i\) and edges \(E_i\).
Examples: social networks, trade networks.
Key challenges: Size and topology vary across graphs. Dependence structure is intrinsic. Graphs encode relational information, not just attributes.
Typical tools:
A dataset is high‑dimensional when the number of variables \(p\) is large relative to the number of observations \(n\): \[ p \ge n \quad \text{or} \quad p \gg n. \]
This is often summarized as ‘large \(p\), small \(n\)’ (Johnstone and Titterington 2009).
Very often, complex data is also high-dimensional:
These issues motivate dimension reduction and regularization (Bühlmann and Van De Geer 2011).
| Aspect | Big Data | High‑Dimensional Data |
|---|---|---|
| Main challenge | Computation | Statistical inference |
| Typical \(n\) | Very large | Small to moderate |
| Typical \(p\) | Moderate | Large |
| Key risk | Scalability | Overfitting |
| Typical tools | Distributed systems | PCA, sparsity |
Some methods in this course apply beyond the strict \(n \gg p\) case, others have a limited interpretation when \(p\) rises.
Univariate statistical analysis focuses on the description of a single variable (\(p=1\)), through a finite sample of \(n\) observation.
Graphical representations: Graphical tools are part of data analysis as they provide intuition about the data set before formal analysis.
Numerical indicators: usually we work on the relative frequencies (e.g. mode).
Graphical representations:
It summarizes the central tendency of the observed data.
It measures the dispersion of the data around the sample mean.
It is expressed in the same units as the variable and is often easier to interpret than the variance.
Bivariate statistical analysis studies the relationship between two variables observed on the same individuals.
The appropriate tools depend on whether the variables are categorical or numerical.
Two categorical variables: stacked or juxtaposed bar charts are used to compare joint and conditional distributions.
Two numerical variables: a scatterplot visualizes the form, strength, and direction of the relationship.
One numerical and one categorical variable: parallel boxplots compare the distribution of the numerical variable across categories.
Graphical representations are an essential first step to detect association, trends, or anomalies.
The relationship between two categorical variables is assessed using the chi-square test of independence, which compares observed frequencies to those expected under independence.
This test evaluates whether the variables are statistically independent.
Covariance measures the joint variability of the two variables but depends on their units.
Correlation is a standardized, unit-free measure of linear dependence.
Remark.
There is no real consensus on the vocabulary around data science, but we can characterise subfields in terms of their respective goals and tools.
| Field | Core Question | Characteristics |
|---|---|---|
| Statistics | What can we infer about the world? | Often model‑based (parametric or not). Emphasis on bias, variance, confidence intervals. |
| Machine Learning | How well can we predict? | Algorithmic and optimization‑driven. Focus on generalization performance. Often trades interpretability for accuracy. |
| Data Analysis | What does the data show? | Exploratory and descriptive. Visualization, summaries, diagnostics. |
| Data Mining | What patterns exist? | Pattern discovery in large datasets. Clustering, association rules, anomaly detection. |
| Artificial Intelligence | How can machines learn from data? | Broader field encompassing ML algorithms and data-driven decision-making. Reasoning, pattern recognition, or adaptation. |
Modern data science blends all of these perspectives into one field (James et al. 2021).
There is an important distinction between:
| Method | Field | Typical Use | Supervised / Unsupervised |
|---|---|---|---|
| PCA | Statistics / Data Analysis | Dimension reduction | Unsupervised |
| LDA | Statistics / Data Analysis | Classification, interpretation | Supervised |
| CART | Machine Learning | Nonlinear prediction | Supervised |
| Bootstrap | Statistics / Machine Learning | Uncertainty estimation | – |
| Bagging | Machine Learning | Variance reduction | Supervised |
| Random Forests | Machine Learning | Robust prediction | Supervised |