Visualizing fitness tracker data

I recently started wearing a fitness band and wanted to visualize the activity data in a heatmap format. This article will go through my thought process in deciding how to parse the data. It is not intended to be a copy and paste tutorial since the fitness band, app and visualization library I choose is likely to be different. One of my key considerations when purchasing devices is the ability to export data into a open format, as this allows me to process it any way I like.

Firstly, I chose to visualize the data using highcharts.js library. The library expects a CSV file in the following format. Each row represents a 10 minute interval. I clocked 38 steps from 12.40pm to 12.50pm and 20 steps from 1 to 1.10pm.

1
2
3
4
5
6

    Date,Time,IndiSteps
    2017-12-16,12.30,0
    2017-12-16,12.40,38
    2017-12-16,12.50,0
    2017-12-16,13.0,20
    2017-12-16,13.10,0

The raw data exported from the fitness tracker app took on the following format. Step data was exported separately from sleep data. The first issue I had to solve was to get the individual steps for each time interval from the cumulative steps logged by the app. This was quickly done using an excel formula =IF(A3=A2,C3-C2,0) which checked to see if the next row was the same date, and if it was, deduct the current step count from the next row's step count.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

    #steps.csv
    Date,Time,CumulativeSteps,IndiSteps
    2017-12-16,12:31:00,0,38
    2017-12-16,12:41:00,38,0
    2017-12-16,12:51:00,38,20
    2017-12-16,13:09:00,58,0
    2017-12-16,13:19:00,58,0

    #sleep.csv
    start,end,type
    13/12/2017 01:39:00,13/12/2017 02:02:00,Light sleep
    13/12/2017 02:02:00,13/12/2017 03:19:00,Deep sleep
    13/12/2017 03:19:00,13/12/2017 03:34:00,Light sleep

Once I was able to retrieve the individual step count for each interval, the next steps were more complicated and required a powershell script. The first step involved rounding off all timestamps to the nearest 10 minutes. I then used the rounded off timestamp as a key and the step count as the value and stored it in a hashtable. sleep.csv required a bit more work. Because the app only exported the start and end time of each sleep, I rounded off the start and end times to the nearest 10 minutes and then started iterating and generating a timestamp every 10 minute. I set the step count to -1 to indicate light sleep and -2 to indicate deep sleep. Using the rounded off timestamp as a key again, I updated the hashtable with the value for sleep intervals.

The reason why I needed to round off to the nearest 10 mins is to ensure the same start and end point for each interval for both steps and sleep. If I had not done so, my hashtable would have entries of -1,0,-1,0 interspersed at 10 min intervals during sleep time. The reason why I chose to overwrite step data with sleep data is because step data existed for the entire 24 hours while sleep data existed for only sleep duration. If I had done it the other way round, all sleep data would have been completely overwritten by step data. A hashset was used because it guarantees only one data point for each interval. Hence, the datapoint for sleep will not be added alongside the one for steps.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

$hashTable = @{}
$times = import-csv "steps.csv"
ForEach ($time in $times){
    $indisteps = $($time.IndiSteps)
    $date = $($time.Date)
    $string = $($time.Time)
    $startdate=[datetime]::ParseExact($date, "dd-MMM-yy", $null)
    $starttime=[datetime]::ParseExact($string, "h:mm:ss tt", $null)
    $newtime = $starttime.AddMinutes(-($starttime.minute % 10))
    $key = $startdate.tostring("yyyy-MM-dd,") + $newtime.tostring("H.m,")
    $hashTable[$key] = $indisteps 
}
$times = import-csv "sleep.csv"
ForEach ($time in $times){
    $start = $($time.start)
    $end = $($time.end)
    $startdate=[datetime]::ParseExact($start, "dd/MM/yyyy HH:mm:ss", $null)
    $startdate.AddMinutes(-($startdate.minute % 10))
    $enddate=[datetime]::ParseExact($end, "dd/MM/yyyy HH:mm:ss", $null)
    $enddate.AddMinutes(-($enddate.minute % 10))
    $tmpdate = $startdate.AddMinutes(-($startdate.minute % 10))
    while($tmpdate -le $enddate){
        $tmpdate = $tmpdate.AddMinutes(10)
        $key =  $tmpdate.tostring("yyyy-MM-dd,H.m,")
        If($type -eq "Light sleep"){
            $hashTable[$key] = "-1"
        }
        ElseIf($type -eq "Deep sleep"){
            $hashTable[$key] = "-2"
        }
        ElseIf($type -eq "Awake"){
            $hashTable[$key] = "0"
        }
    }
}
$hashTable.GetEnumerator() | sort -Property key | Out-File out.csv

One final tweak was required for the charting library to display sleep data distinctly from step data. After all, we do not want -1 and -2 step (which is sleep) to look very similar to 0 step. This was achieved by setting the percentile of the different color stops. 0th percentile which corresponds to -2, or deep sleep will be displayed as black (#000) while 0.002th percentile which is -1, or light sleep will be displayed as grey. Anything onwards will be displayed as blue. The rest of the color stops were unchanged, with yellow for moderate activity (50th percentile) and red for high activity (100th percentile).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

colorAxis: {
    stops: [
        [0, '#000'],
        [0.002, '#535353'],
        [0.003, '#3060cf'],
        [0.5, '#fffbbc'],
        [0.9, '#c4463a'],
        [1, '#c4463a']
    ],
    min: -2,
    max: 497,