Introduction to Data Science¶
1MS041, 2023¶
©2023 Raazesh Sainudiin, Benny Avelin. Attribution 4.0 International (CC BY 4.0)
In [65]:
from Utils import showURL
In [66]:
showURL("https://en.wikipedia.org/wiki/Continuous_uniform_distribution")
Out[66]:
Integrate the uniform distribution using Sympy and simulate¶
In [67]:
from sympy import var, integrate
In [68]:
a = var('a')
b = var('b')
x = var('x')
In [72]:
f=1/(b-a)
In [76]:
F_prim = integrate(f,x)
In [77]:
# We need that F(a) = 0
# F(a) = F_prim(a) + C
# C = -F_prim(a)
F = F_prim - a/(b-a)
F
Out[77]:
We basically start by solving for $y$ $$ F(x) = y $$ This is solvable because F is strictly increasing at least in the interval $x \in (a,b)$.
$$
F(x) = y
$$$$
\frac{x-a}{b-a} = y
$$
$$
F^{-1}(y) = y(b-a)+a
$$
In [78]:
import numpy as np
y = np.random.uniform(0,1,10000)
x = y*(20-10)+10
In [79]:
import matplotlib.pyplot as plt
_=plt.hist(x)
Inversion sampling¶
Inversion sampling starts with a distribution function $F$. We want to simulate from $F$. We first compute $F^{-1}(y)$ and then we let $X$ be a uniform $(0,1)$ random variable and then we define $$ Y = F^{-1}(X) $$ then $Y$ has distribution function $F$.
$$
F_Y(y) = P(Y \leq y) = P(F^{-1}(X) \leq y) = P(X \leq F(y))
$$
Rename $F(y) = x$ then $$ P(X \leq x) = F_X(x) = F_X(F(y)) $$
Since $X$Â is uniform $(0,1)$ we have that $F_X(x) = x$ if $x \in (0,1)$
Conclusion is that $F_Y(y) = F(y)$.
Integrating the exponential and inversion sampling¶
In [80]:
showURL("https://en.wikipedia.org/wiki/Exponential_distribution")
Out[80]:
In [81]:
lam = var('lambda')
x = var('x')
from sympy import exp
f = lam*exp(-lam*x)
f
Out[81]:
In [83]:
F_prim = integrate(f,x)
In [84]:
F_prim
Out[84]:
In [ ]:
In [85]:
F = F_prim+1
F
Out[85]:
$$
F(x) = y
$$$$
1-e^{-\lambda x} = y
$$$$
1-y = e^{-\lambda x}
$$
$$
\ln(1-y)/(-\lambda) = x
$$$$
F^{-1}(y) = \frac{\ln(1-y)}{-\lambda}
$$
In [86]:
y = np.random.uniform(0,1,10000)
x = np.log(1-y)/(-1)
In [89]:
_=plt.hist(x,bins=100,density=True)
x_plot = np.linspace(0,9,100)
plt.plot(x_plot,np.exp(-x_plot))
Out[89]:
Loading the co2 data and plotting¶
In [90]:
%%bash
ls data
In [93]:
%%bash
head -n 100 data/co2_mm_mlo.txt
In [94]:
with open("data/co2_mm_mlo.txt",mode="r") as f:
current_line = f.readline()
while (current_line[0] == '#'):
current_line = f.readline()
print(current_line)
In [99]:
import csv
data_raw = []
with open("data/co2_mm_mlo.txt",mode="r") as f:
current_pos = f.tell()
current_line = f.readline()
while (current_line[0] == '#'):
current_pos = f.tell()
current_line = f.readline()
f.seek(current_pos)
csv_reader = csv.reader(f,delimiter=' ',skipinitialspace=True)
for line in csv_reader:
data_raw.append(line)
In [100]:
data_raw[0]
Out[100]:
In [101]:
len(data_raw)
Out[101]:
What is a schema?¶
In [102]:
schema = [int, int, float, float, float, float, int]
In [104]:
type(int("123"))
Out[104]:
In [105]:
data_parsed = [[typ(item) for item,typ in zip(line,schema)] for line in data_raw]
In [106]:
data_parsed[0]
Out[106]:
Loading data into a numpy array and types¶
In [107]:
data_array = np.array(data_parsed)
In [110]:
data_array.shape
Out[110]:
In [111]:
data_array[0,0]
Out[111]:
Some basic stats using the Utils package¶
In [112]:
# Basic stats
from Utils import basic_stats
basic_stats(data_array[:,4])
In [117]:
data_array[0:10,4]
Out[117]:
In [119]:
x = np.random.normal(0,27.53,10000)
basic_stats(x)
In [124]:
_=plt.hist(data_array[:,4],bins=50)
In [125]:
# EDF
from Utils import makeEDF, plotEDF
plotEDF(makeEDF(data_array[:,4]))
