The point/select task is still the core of the current graphical user interfaces. An average user executes way over a thousand point/select tasks a day. point.. click.. point.. click.. This pointing work you do is not trivial. Often, your cursor shoots over the intended target. You are then forced to make a corrective movement to reach the target. The system does not know or speculate upon your intentions before you have clicked the target. You have to be precise.

Think of it in tennis: A tennis player does not wait to think about where its opponent tries to hit the ball until the ball has landed in his part of the court. Instead the player anticipates. By analyzing the position and movements of its opponent as well as the course of the ball right after it has left the opponents racket, the tennis player anticipates where on his part of the court the ball will land. Selecting a target could be made easier if your computer knew sooner where you want to point your mouse. The system could take a more active role in getting there. Is it possible for the system to know what target you try to reach before you reached it? Could be. The beginning of your mouse movement might contain enough information to determine what target you intend to reach.

Join our Experiment

Right now, we are collecting mouse movement of real users. We need them to test our algorithm. You can donate your mouse movements by running the experiment. In the experiment you will be a asked to click one hundered targets. It will only take three minutes. You can also test and watch our current predictor algorithm in action in our demo.

How we do it: Genetic Algorithms

We try to find a target prediction algorithm that determines the endpoint of your mouse movement by analyzing the beginning. Finding a target prediction algorithm is largely an optimalisation problem. Optimizing the best prediction from possible manipulations on speed, velocity, and direction. This kind of optimalisationproblems can be solved with the help of genetic algorithms. A genetic algorithm is a computer program that tries potential solutions and evaluates them. The best solutions are combined and tested again, the best of these kids can grow offspring again. This cycle is repeated until no better solutions arise. Simple prediction strategies can be coded in strings of symbols, we call these chromosomes. Every symbol in a chromosome gives a weight to the prediction strategy. A set of chromosomes is defined, these are tested on their accuracy. Chromosomes with a high fitness may propagate, others die out. Propagation goes like in nature, paring strings exchange material. This is called crossover. Crossover is important, but it only mixes things up and does not bring in new material. This is done by adding some noise (mutation) in de duplication process.

1. User moves mouse towards target. Predictors are analyzing the mouse movement. 2. The predictors estimate the endpoint of the mouse movement based upon the movement so far. 3. A predictor with a bad reputation is deleted from the population. 4. The two best predictors are allowed to combine chromosomes and grow offspring. 5. The kid (new predictor) is incorporated in the population of predictors. 6. step 1 to 5 are repeated for every target that is pointed. 7. In time, a stable population of good target predictors evolves.

= target = cursor = good predictor = bad predictor = average predictor

Above is shown how the genetic algorithm works for our target prediction problem. A population of target predictors is analyzing the mousemovements of a user that is clicking predetermined targets (1). Once the first part of the users mouse movement is executed (usually right after the mousespeed has reached its maximum) the predictors make their guesses about the endpoint of the movement and are evaluated (2). Some predictors have a good reputation, others not. The predictor with the worsed record of targed predictions is removed from the population (3). Its place is taken by a fresh predictor. The new predictor is composed by combining the two predictors with the best reputation. In this process a little random noise is added as well (4 and 5). This process is repeated for every target that is clicked. In time the population stabilises. We end up with an optimal mouse trajectory predictor for this user.

Contact

Want more information about this research or actually talk back? Contact Koert van Mensvoort (email) or Hilde Keuning (email)
Want to learn more about genetic algorithms? Here is an intuitive introduction to the subject of genetic algorithms.