Dataset and Game Description

Link to YouTube video of task demo

Figure 1: The Lost In Migration Game Interface. The apperance of single trial of the game. The user responds by indicating the direction of the central target bird (in this case - up).

The dataset we received had already undergone some filtering, retaining only users who completed at least 5 game sessions, and only users who completed at least 99% of their game sessions on the browser-based web version of the game (as opposed to the mobile version). The total dataset consists of 8,551 individual users, 168,307 separate game sessions, and 8,207,980 total trials of the game. The demographic data included with each user consists of the age, gender, and highest education level achieved. The data set also includes a global timestamp for each game session, which will allow us to control for the influence of temporal spacing between game sessions. Such considerations may be important for reducing noise in our model predictions, due to the real-world reality of users having full control over the timing of each game session (e.g. the first 10 gameplays could occur on the same day, or spread out over a month).

Figure 2: The seven spatial layouts that the target and flankers can appear in. The layouts vary trial to trial, and all seven layouts are present in both the baseline and split-test versions of LIM. In all cases, the task is to respond with the direction of the central bird

Each game of LIM has a set duration of 45 seconds, during which users complete as many trials as can fit within that time frame. Figure 1 illustrates the screen view for a trial of the game. On each trial, users are presented with a configuration of 5 “birds” - a central “target” bird and 4 “flanker” birds. The task is to indicate the direction of the central bird as fast as possible using the up/down/right/left arrow keys on their device. The task parameters that define each trial include the direction of the target bird, the direction of the flanker birds, the XY coordinates of the target bird on the screen, and the spatial layout of the birds (Seven layouts - shown in Figure 2). On 50% of trials, the flanker birds are pointing the same direction as the target birds (congruent), while on the other half of trials the flanker birds point in any of the 3 incongruent directions. The dependent measures recorded on each trial are the reaction time, accuracy (correct or incorrect response), response direction (up,down,right,left), and the gamified performance score shown to the users (a combination of reaction time and accuracy). Figure 3 shows the rapid improvements that occur in the early stages of the task and also indicates the fairly large effect of user age on performance.

Figure 3: Learning curves over the first 200 trials. Users are separated into bins according to their age. Note that these smooth, aggregated curves are not necessarily represenatitve of the typical individual user

Split-Test Data

A subset of the users in the dataset participated in an internal experiment or “split-test” conducted by Lumos Labs, wherein new users could be randomly assigned to either the standard version of the game (described above) or to one of several modified game versions. The split-test versions of LIM each expanded the range of possible values for a dimension of the task, see 4. Note that users could be assigned to a version of the split test that included one, two, or all three of the expanded dimensions.

Expanded dimensions of the split test

• Rotation and Orientation version - A continuous range of discrepancies between target and flanker birds on incongruent trials ( compared to fixed to 90, 180, or 270 degrees in base version)
• Distance between birds version - A 0-20px continuous distance between target and flankers (constant value of 41px in base)
• Size of bird version - The size of the target bird and flankers birds (same size for all 4) are independently drawn from a range of 35-60px (both fixed at 41px in base)

Figure 4: Illustration of expanded dimensions in the split-test versions fo the game.Users could be assigned to a version of the split test that included one, two, or all three of the expanded dimensions. (note that orientation and rotation are manipulated together). Distance between flankers manipulation not shown

At the start of each new game, Split-test users had a small probability of being permanently switched to the standard version of the game. Figure 5 shows the number of users in the baseline or split-test version of the game over the first 10 games. Figure 6 separates users who only experience the baseline version from users who start with the split-test, and eventually switch to the baseline version. A small number of users who begin with the baseline version are later assigned to a single game of a split-test version, however the frequency of such users may be too low to be useful for the present purposes.

Randomization

As mentioned above, the trial selection process is not entirely-random, due to the constraint congruent and incongruent trials occur at approximately equal proportions. However, all other aspects of the trial generation process are random (i.e. no dependence on previous states, user performance, number of gameplays). A simple consequence of such randomization, is that some users will experience a wider range of trial-states, particularly in the early stages of the game. Figure 7 illustrates a toy case wherein two users receive four trials with discrepant levels of variation in spatial layout, the XY coordinates of the birds on the screen, and bird direction.

Figure 7: A) Example of a sequence of first four trials with relatively high variation, in terms of spatial layouts, bird directions, and position on the screen. B) Example of a sequence of first four trials with lower variation in the aforementioned dimensions.

Measuring Trial-by-trial variability

To quantify the amount of variability experienced by a user, we may start by simply taking the number of unique trial-states that the user has encountered after a given number of trials with the game. Each trial of the game can be defined a long many different dimensions , both discrete (e.g. bird direction, layout), and continuous (X and Y coordinates on the screen). For simplictly, consider only the 3 categorical/ordinal trial dimensions, which consist of 4 values for target direction, 4 for flanker direction, and 7 spatial layouts, resulting in 112 (\(4*4*7\)) distinct trial-states. Figure 8 demonstrates how users who have completed the same total number of trials will still differ in the number of unique trial-states experienced.

Although nice for its simplicity, quantifying variability as the number of unique states experienced is a fairly coarse and limited metric. One limitation comes through the lack of spatial relations between trial-states, for example n 7 the lower variation example in panel B consists of four unique trial-states, but neverthless covers a far narrower region of the full state-space compared to the high variation example in panel A. Another issue, reflected in 8, is that given enough trials, the differences between users will eventually become negligible as they each approach the maximum of 112 unique states encountered. A more suitable measure may therefore be uniformity of the frequency distribution of trial types (e.g. a measure of entropy).

Similarity Between Trials

A central challenge of project 3 will be to establish an appropriate measure of distance between the current trial \(trial_n\) and prior experience \(trial_{1:n-1}\). Similarity could be defined as the simple dimensional/featural overlap between two trials, or by modelling a trial-state as a point in a multidimensional space, and using some distance metric (e.g., Euclidean distance) to compute the distance between trials, and then transforming that distance into psychological space with an exponential or Gaussian function.

As was described above, each trial can be defined a long a number of dimensions. However are unlikely to be equal in their influence, and may thus need to be differentially weighted.Determining the identies of the dimensions themselves may also be nontrivial, for instance whether the full set of 16 combinations of target direction (up,down,right,left), and flanker directions (up,down,right,left) can be simplified into a single metric of flanker rotation relative to target location,e.g. 0 , 90 , 180 and 270 .

Once a similarity measure between trials has been established, we can use the individual trial histories for each user to compute how similar a given trial is to the totality of their prior experience in the game (i.e., the summed similarity of all previous trials), or to the similarity of a subset of their more recent trials. We can then begin to perform inferential statistics assessing the extent to which users matched in total experience with the game may perform differently on individual trials as a function of the similarity between their prior experience and that particular trial. Additionally, our similarity metric can be used to attempt to explain differences in performance between baseline, split-test, and split-test -> baseline users.

Measurement model of learning and performance

A prerequsite for assessing the influence of variability/similarity on the trial to trial level is to have an adequate measure of trial-level performance. Individual reaction times, the primary performance measure of the task, are likely far too noisy to be suitable for this purpose. Instead of modelling raw reaction times, we can instead attempt to model the latent level of skill the user has for a task as some function of the number of trials completed. Two of the most widely used functions for relating number of trials completed to performance are the exponential and power functions (see Equation 1 and Equation 2). Both functions are commonly fit with three free parameters, \(u\), \(a\) and \(c\), which reflect starping performance, asymptotic performance, and learning rate, respectively. The debate between these two learning functions is longstanding and continues to this day (Evans et al. 2018; Heathcote, Brown, and Mewhort 2000). Previous research using data from the base verion of LIM did find a slight advantage for the power model(Steyvers and Benjamin 2019). However the investigation of Steyvers included data from a far larger number of games from each user than what the proposed project will consider (due to the constraint of when sufficient between-user trial variability occurs ref/figure). We thus intend to test the suitability both learning functions. With an appropriate model of learning perforance obtained for each user, we will then be able to evaluate the ability of our measures of variability and similarity expereinced up to \(trial_{n-1}\) at predicting performance at on \(trial_n\).

Exponential Model: \(y_t = u - ae^{-ct}\)

Power Model: \(y_t = u - at^{-c}\)