# Descriptive statistics

### Foreword

I have worked with statistics one way or another ever since University. I also happen to think that statistics are kind of a big deal, so decided to put some posts together about them. This post concentrates on descriptive statistics, but I want to cover other areas in later posts. You can also download this post as a PDF if you like.

Before I launch in to the main body of the post it is worth noting that all of the following descriptions, equations and implementations assume that measurements are taken from a statistical sample (i.e. a subset of the population).

### Mean

The sum of all measurements divided by the count of all measurements. The mean is well suited when a sample has a normal distribution. It is influenced by outliers so cannot be considered robust.

##### Math

where $\bar{x}$ is the arithmetic mean; $n$ is the count of all measurements and $x_i$ is an individual measurement.

##### Scala Implementation
def mean(seq: Seq[Double]): Double = {
seq.sum / seq.length
}


### Median

The measurement separating the lower half of a sample from the upper half. If there is an even number of measurements then the median is defined as the mean of the two middle values. The median is well suited when a sample has a skewed distribution. It is not influenced by outliers so can be considered robust.

##### Math

where $\tilde{x}$ is the median; $n$ is the count of all measurements; $x$ is a sequence of ordered measurements and $x_i$ is an individual measurement.

##### Scala Implementation
def median(seq: Seq[Double]): Double = {
val n = seq.length
if (n % 2 == 0) (seq(-1 + n / 2) + seq(n / 2)) / 2
else seq(n / 2)
}


### Standard Deviation

The standard deviation provides an approximation to the average absolute deviation of a measurement from the mean. Intuitively it gives an indication of the spread of a sample. It is sometimes used to identify outliers.

##### Math

where $\sigma$ is the standard deviation; $n$ is the count of all measurements; $x_i$ is an individual measurement and $\bar{x}$ is the mean.

##### Scala Implementation
def std(seq: Seq[Double]): Double = {
val n = seq.length
val m = mean(seq)
sqrt(seq.map(x => pow(x - m, 2)).sum / (n - 1))
}


### Coefficient of Variation

The standard deviation is an absolute measure of variability. To compare the variability of two or more samples a relative measure is required: the coefficient of variation.

##### Math

where $c_v$ is the coefficient of variation; $\sigma$ is the standard deviation and $\bar{x}$ is the mean.

##### Scala Implementation
def cv(seq: Seq[Double]): Double = {
std(seq) / mean(seq)
}


### Skewness

Skewness measures the degree of asymmetry exhibited by the sample data. If skewness equals zero then the sample distribution is symmetric about the mean. The adjusted Fisher-Pearson standardized moment coefficient is shown here (used by Excel, Minitab and SPSS).

##### Math

where $G$ is skewness, $n$ is the count of all measurements; $x_i$ is an individual measurement; $\bar{x}$ is the mean and $\sigma$ is the standard deviation.

##### Scala Implementation
def skew(seq: Seq[Double]): Double = {
val n = seq.length.toDouble
val m = mean(seq)
val s = std(seq)
(n / ((n - 1) * (n - 2))) * seq.map(x => pow((x - m) / s, 3)).sum
}


### Kurtosis

Kurtosis measures how peaked the sample data is. Peaked data has a kurtosis value greater than three. Flat data has a kurtosis value less than three.

##### Math

where $G_2$ is kurtosis, $n$ is the count of all measurements; $x_i$ is an individual measurement; $\bar{x}$ is the mean and $\sigma$ is the standard deviation.

##### Scala Implementation
def kurt(seq: Seq[Double]): Double = {
val n = seq.length.toDouble
val m = mean(seq)
val s = std(seq)
val a = (n * (n + 1)) / ((n - 1) * (n - 2) * (n - 3))
val b = seq.map(x => pow((x - m) / s, 4)).sum
val c = (3 * pow((n - 1), 2)) / ((n - 2) * (n - 3))
(a * b) - c
}


### Standard Scores

Standard scores (z scores) may be interpreted as the number of standard deviations a measurement is away from the mean.

##### Math

where $z_i$ is a standard score; $x_i$ is an individual measurement; $\bar{x}$ is the mean and $\sigma$ is the standard deviation.

##### Scala Implementation
def zScores(seq: Seq[Double]): Seq[Double] = {
val m = mean(seq)
val s = std(seq)
seq.map(x => (x - m) / s)
}


# The cake pattern

Scala's 'explicitly typed self references' permit compile-time dependency injection when using traits and classes with no-argument-constructors. Given that this is quite a useful trick, the community decided to give it a name: the cake pattern.

Let us assume we have the following service orientated architecture:

Where Customer is a case class containing a list of Account (also a case class) objects; CustomerService and AccountService are both abstract traits that lookup Customer and Account instances respectively; NaiveCustomerService and NaiveAccountService are traits providing simple in-memory implementations. Here is the code:

case class Customer(id: Long, name: String, accounts: List[Account])

case class Account(name: String, balance: BigDecimal)

abstract trait CustomerService {
def customers(id: Long): Customer
}

abstract trait AccountService {
def accounts(customerId: Long): List[Account]
}

trait NaiveAccountService extends AccountService {
private val accountsById = Map(
1l -> List(Account("Current", 500), Account("Savings", 900)),
2l -> List(Account("Current", 450)))

def accounts(customerId: Long): List[Account] = accountsById(customerId)
}

trait NaiveCustomerService extends CustomerService { self: AccountService =>
private val namesById = Map(
1l -> "Matt Roberts",
2l -> "Joe Bloggs",
3l -> "John Doe")

def customers(id: Long): Customer = Customer(id, namesById(id), accounts(id))
}


The case classes, abstract traits and even the NaiveAccountService trait should look familiar. The first use of an explicit self type is in NaiveCustomerService trait. The statement self: AccountService => declares that the NaiveCustomerService trait depends on an AccountService without defining an implementation (NaiveAccountService for example). You can see why that dependency exists on the penultimate line where the method accounts(id) is called.

Using the NaiveCustomerService is now just a matter of mixing in the NaiveCustomerService and an associated AccountService (I use NaiveAccountService for simplicity):

object Cake extends App with NaiveCustomerService with NaiveAccountService {
val customer = customers(1)
printf("Name: %s\n", customer.name)
customer.accounts.foreach(account => printf("%s: %.2f\n", account.name, account.balance))
}


If you have lots of services then you will probably want to instantiate them separately or in groups. Scala provides a nice syntax for this:

object Cake {
class MyService { self: CustomerService => }
def main(args: Array[String]) {
val service = new MyService with NaiveCustomerService with NaiveAccountService
val customer = service.customers(1)
printf("Name: %s\n", customer.name)
customer.accounts.foreach(account => printf("%s: %.2f\n", account.name, account.balance))
}
}


The one major drawback to the cake pattern is that if you wish to mock services (using say Mockito) then you will need to use the provider pattern. If you have a lot of services then you'll have an equal number of providers. This can become quite unwieldy, so you may prefer the more traditional many-argument-constructor approach.

The complete project can be found on GitHub.

# Add a swap file to an EC2 micro instance

It turns out that an EC2 micro instance does not come with a swap file. If I had been paying attention I would have noticed that top reports this fact quite clearly and acted straight away. As it happens I only worked this out after my java process kept getting killed by the operating system (thanks to dmesg). Here's how I added a swap file to that instance.

First I became root:

sudo su -


Second, I created a swap file using dd:

dd if=/dev/zero of=/swapfile bs=1M count=1024


Where if is the input file, of is the output file, bs is the bytes and count is the number of bytes.

Third, I set up that file as a Linux swap area using mkswap:

mkswap /swapfile


Fourth, I enabled the file for paging and swapping using swap on:

swapon /swapfile


NB: At this point you can run top and see the swap in action.

Fifth, I set the swap file to mount on startup by adding this line to the end of /etc/fstab:

/swapfile    swap    swap    defaults    0    0


# Scala 2.10 reflection and serialization

Scala 2.10 includes a reflection library. It's still considered experimental so you have to explicitly add it in your build.sbt file:

scalaVersion := "2.10.1"

name := "reflection"

libraryDependencies += "org.scala-lang" % "scala-reflect" % "2.10.1"

libraryDependencies += "com.fasterxml.jackson.core" % "jackson-core" % "2.1.4"


One nice feature is that it lets you discover the names of constructor parameters at run time. That's helpful if you want to serialize / deserialize immutable classes. Here's some example serialization code:

import com.fasterxml.jackson.core.{JsonGenerator, JsonFactory}
import java.io.StringWriter
import scala.reflect.runtime.universe._

object App {
case class Person(name: String, age: Int, hobbies: List[String])

def main(args: Array[String]): Unit = {
println(toJson(new Person("Matt", 26, List("biking", "guitar", "helli skiing"))))
println(toJson(List(
new Person("Matt", 26, List("biking", "guitar", "helli skiing")),
new Person("Joe", 25, List("swimming", "piano", "deep sea diving")))))
}

def toJson(a: Any): String = {
val f = new JsonFactory()
val sw = new StringWriter()
val g = f.createGenerator(sw)
serialize(a, g)
sw.toString
}

private def serialize(a: Any, g: JsonGenerator): Unit = {
a match {
case a: Byte => serializeValue(a, g)
case a: Short => serializeValue(a, g)
case a: Int => serializeValue(a, g)
case a: Long => serializeValue(a, g)
case a: Float => serializeValue(a, g)
case a: Double => serializeValue(a, g)
case a: Boolean => serializeValue(a, g)
case a: String => serializeValue(a, g)
case a: Seq[_] => serializeArray(a, g)
case _ => serializeObject(a, g)
}
g.flush()
}

private def serializeValue(a: String, g:JsonGenerator): Unit = {
g.writeString(a)
}

private def serializeValue(a: AnyVal, g: JsonGenerator): Unit = a match {
case a: Byte => g.writeNumber(a)
case a: Short => g.writeNumber(a)
case a: Int => g.writeNumber(a)
case a: Long => g.writeNumber(a)
case a: Float => g.writeNumber(a)
case a: Double => g.writeNumber(a)
case a: Boolean => g.writeBoolean(a)
}

private def serializeArray(a: Seq[_], g: JsonGenerator): Unit = {
g.writeStartArray()
a.foreach(a => serialize(a, g))
g.writeEndArray()
}

private def serializeObject(a: Any, g: JsonGenerator): Unit = {
val aMirror = mirror.reflect(a)
val aType = aMirror.symbol.typeSignature
val ctor = aType.declaration(nme.CONSTRUCTOR).asMethod
val ctorSymbols = ctor.paramss.head.map(p => aType.declaration(p.name))
val ctorTerms = ctorSymbols.map(s => s.asTerm)
val ctorFieldMirrors = ctorTerms.map(t => aMirror.reflectField(t))
val ctorValues = ctorFieldMirrors.map(f => f.get)
g.writeStartObject()
ctorSymbols.zip(ctorValues).foreach((tuple) => {
val (key, field) = tuple
g.writeFieldName(key.name.decoded)
serialize(field, g)
})
g.writeEndObject()
}
}