Thomas Kuebler – Perception Engineer

Tag: Eye-Tracking

  • Google Colab Webcam streaming and processing

    One of the code examples that Colab provides handles recording of an image via a webcam. The code writes a webcam frame to an image file and displays it. However, it does not directly handle life streaming of the webcam feed.

    I wanted to perform deep learning-based eye-tracking within Colab using the webcam. That would require continuous processing of the video stream. The code to do so can be found here: https://colab.research.google.com/gist/kueblert/64299582829c35a5579b520e0bdfa9a8/cameraaccess.ipynb

    In short, it uses the same interface as the example to pass the frame via javascript from the browser to the Colab Runtime. Therefore, the frame data is converted to a Data URL, i.e., compressed as a jpeg and base64 encoded.

    The python backend decodes the image string and decompresses the jpeg data.

    As I wanted to visualize the data as well, passing data from python to the javascript caller is also demonstrated.

    A word of caution: While the approach does work, the performance is clearly not good and the number of frames you are able to process per second is very limited.

  • Thesis done :-)

    Thesis done 🙂

    My doctoral thesis on Scanpath Comparison Algorithms is published, you can grab a free copy at http://hdl.handle.net/10900/74458 or the compressed e-book version for online reading (with slightly less beautiful figures) at my university webpage.

  • Eye-tracking reveals the Pros in Mario Kart

    Eye-tracking reveals the Pros in Mario Kart

    Ever wondered how your eyes perform while playing a videogame? Turns out they might be quite important for you winning the game. Actually, so important that one can even distinguish whether you are a winner based solely on the movement of your eyes! And here is, what eye-tracking whilst gaming looks like:

    The small blue dots are called fixations, i.e. the spots where the eye rests at (even if it does so only shortly). During a fixation we actually perceive visual information. Fixations are interconnected by saccades, very very fast movements of the eye. In fact, they are so super-fast that our brain suppresses visual perception while we perform them. Try it with a mirror – you won’t be able to see your eyes moving when you look from one spot to the other.

     

    Eye-tracking data comparison

    So when you do an eye-tracking experiment, the result is just a list of fixation locations (and durations) and the saccades in-between. May look fancy on youtube, but doesn’t actually tell you anything meaningful most of the time. What you need to do is calculate some key metrics (such as the average time spent looking at the same location before shifting gaze; or the gaze density at certain game objects).

    Comparing eye movement sequences to each other as a whole (without the restriction to one specific key metric) is non-trivial (and I will likely cover this in a future post, as this is my PhD topic 😉 ). But if we do so, turns out that we can separate good players from novices quite well (Figure 1). It’s not just reactions that we train and getting to know the game better – but also a training effect in the patterns of how we need to move our eyes, that make a good player.

    The percentage of eye-tracking recordings that were correctly classified as either fast or slow drivers are shown on the diagonal. Off-diagonal elements (left top and bottom right) are misclassifications.
    Figure 1: The percentage of scanpaths classified correctly as either fast or slow drivers is shown on the diagonal. Off-diagonal elements (left top and bottom right) are misclassifications.

    T. C. KĂźbler, C. Rothe, U. Schiefer, W. Rosenstiel, E. Kasneci (2016): SubsMatch 2.0: Scanpath comparison and classification based on subsequence frequencies. Behavior Research Methods:1-17