Lia Martinez

Blink Stories is based on a theory that I had wanted to test for a very long time.

It is Walter Murch’s theory about why we blink, our stream of consciousness, and film editing.

I first mention it in this blog here. This is what I wrote:

The theory is that when you blink, there is a switch in your consciousness. What I like about this is that it speaks of empathy; when you watch a film and you blink at the same time as a character, you empathize with them. When you react in the same way as a theater full of people, you are all connected in this way.

For my Data Representation final with Jer Thorp, I decided to explore this theory on my own at last.

These are the questions that I wanted to answer:

1. Do people blink at similar, precise times while watching a movie? Is this proof of a job well done on the filmmaker’s part, in that the audience empathizes wholly with the film?
2. Does it coincide with any of Murch’s markers — while an actor himself blinks, or during an editor’s cut?
3. Is there a particular genre where people empathize particularly strongly with the character? Comedy, drama, horror?

What I particularly find wonderful about Murch’s theory is that when we blink, we ourselves are making a “cut”. When we change consciousness, when we switch our train of thought, we are making a “cut”, just like a film editor does. So we can apply this theory to real life in a number of ways –

1. Do two people blink in similar times during a conversation?
2. What about when people listen to music? What about different types of music — jazz or classical?

Given the limited amount of time, I scaled down the idea and separated it into manageable components:

1. Data Collection
   • Technology: Blink Detection Using Open CV and Processing
   • Write/ Read To XML
   • Amazon Turk to collect a large amount of data
   • A booth in the Tisch Building to collect more data
   • Film Selection: clips from which films?
2. Data Representation
   • A movie poster with every frame in the video clip represented
   • A live representation of the blinks playing back through a video screen.

DATA COLLECTION

BLINK DETECTION USING OPEN CV AND PROCESSING

Open CV is well-loved in ITP because it comes built-in with some neat face-tracking functions. The trick was to find a way to just track the eyes. I finally found this site that carried Custom Haar Cascades, which comes in XML form and basically does the trick of finding the eyes for you. After you have downloaded XML, a few modifications to the file itself and the OpenCV Javadoc is all you need to do.

I was able to find eye Haar Cascades here, with instructions on how to use it with processing here.

Ideally, I would have wanted to use as accurate technology as possible with the blink detection, and looked for a while into DanO’s pupil detection code, as well as consulting with Dan Shiffman and Greg Borenstein. I also looked into the Eyewriter, which I will look into some more in the future. After a few days of experimenting with pupil detection and feeling overwhelmed, I decided that I would need more time to make it work properly and given the time constraints, would need to scale down for the meantime. So for now the code uses a combination of eye-tracking and motion detection — if any motion is detected within the eyes, then call that a blink.

This means that essentially, the viewer cannot move at all during the experiment (or at least, cannot move his/ her head). Even eyeball movement might be counted as a blink, so I decided to make the size of the screen small to minimize this.

void readBlinks() {
  opencv.read();                  
 
  Rectangle[] faces = opencv.detect( 1.2, 3, OpenCV.HAAR_DO_CANNY_PRUNING, 10, 10 );
 
  if (debugOn) image( opencv.image(), 0, 0 );  //  Display the difference image
 
  opencv.absDiff();                           //  Creates a difference image
  opencv.convert(OpenCV.GRAY);                //  Converts to greyscale
  opencv.blur(OpenCV.BLUR, 3);                //  Blur to remove camera noise
  opencv.threshold(thresh);                       //  Thresholds to convert to black and white
  movementImg = opencv.image();               //  Puts the OpenCV buffer into an image object
 
  // draw face area(s)
  noFill();  
  stroke(255, 0, 0);
 
  for ( int i=0; i<faces.length; i++ ) {
    if (debugOn) rect( faces[i].x, faces[i].y, faces[i].width, faces[i].height );    // draw the rectangle
 
    for (int x = faces[i].x; x < (faces[i].x+faces[i].width); x++) {      //loop through 
      for (int y = faces[i].y; y < (faces[i].y + faces[i].height); y++) {         
             if (isPlaying) {                     
                   if (blinkTime != oldTime && blinkTime - oldTime > interval) {
                        if (brightness(movementImg.pixels[x+(y*movementImg.width)]) > 230) {  //if brightness is higher
                        totalPixels++;
            }
 
            if (totalPixels >10) {
              addNewBlink(blinkTime);
              saveData();
              oldTime = blinkTime;  
              totalPixels = 0; 
 
              if (debugOn) fill (255, 0, 0); 
              if (debugOn) ellipse (100, height/2 - 100, 30, 30);
              println ("MARKER " + blinkTime);
            }
          }
        }
      }
    }
  }
  opencv.remember(OpenCV.SOURCE);
}

XML WRITE/ READ WITH PROXML

The XML Writing/ Reading was the part that I struggled with the most, probably because I chose this particular time to learn all about ArrayLists. The XML Tree was simple enough:

<User>
<Clip num = “1”>
<blink time = “15.01”>
<blink time = “22.00”>
<blink time = “42.45”>
</Clip>
</User>

So basically every user would have a clip, and every clip would have a blink with their corresponding times.

The recorder application I wrote has the ability to create a new User, and within that user, new Clips, and within those Clips, blinks.

First I create a new user every time the program starts up, in setup:

  newUser = new proxml.XMLElement ("user");
  userNum++;
  newUser.addAttribute ("num", userNum); 
  XMLusers.addChild(newUser);

New clips are created whenever a new movie is loaded, with keypress:

  else if (key == '1') {
    if (loaded) movie.pause(); 
    movie = new Movie (this, "lilmermaid.mp4"); 
    blinkTime = 0; 
    oldTime = 0; 
    movie.play();
    movie.goToBeginning();
    movie.pause();
    loaded = true; 
    println ("loaded Little Mermaid"); 
    newClip = new proxml.XMLElement ("clip");
    newClip.addAttribute ("num", "1");
    newUser.addChild(newClip);
    state = 2;
  }

New blinks are created with a function “addNewBlink” that is called when OpenCV finds a new blink:

void addNewBlink (float time) {
  blinks.add (new Blink(time));
 
  proxml.XMLElement blink = new proxml.XMLElement ("blink"); 
  blink.addAttribute ("time", time);
  newClip.addChild(blink); 
  //newUser.addChild(blink);
}

This is called whenever an XML is loaded. Number of users already in the XML is counted, so that when a new user is created it increments by 1.

void xmlEvent(proxml.XMLElement element) {
  XMLusers = element;
 
  proxml.XMLElement[] users = XMLusers.getChildren();
  userNum = users.length;
  println ("User Size: " + users.length);
}

The generator applications then read the final XML:

void xmlEvent(proxml.XMLElement element) {
  XMLusers = element;
 
  //create array of proxml elements called "users", get the children
  proxml.XMLElement[] users = XMLusers.getChildren();
  for (int i = 0; i < users.length;i++) 
  {
    User u = new User();
    u.num = users[i].getIntAttribute("num");
    println ("------new user------");
 
    //create another array of the children of the users
    proxml.XMLElement[] clips = users[i].getChildren(); 
    for (int j = 0; j < clips.length; j++) {  
      Clip c = new Clip();    
      c.whichUser = u.num; 
      c.num = clips[j].getIntAttribute ("num"); 
      println ("--new clip -- " + c.num); 
 
      //get the blinks within the clip array
      proxml.XMLElement[] blinks = clips[j].getChildren(); 
      for (int q = 0; q < blinks.length; q++) {  
 
        float blink = blinks[q].getFloatAttribute ("time");
        //add the blink to the arraylist within the arraylist!
        c.blinks.add(blink);
      }
 
      u.clips.add(c);
    }
    // add it to the arraylist
    userList.add(u);
  }
}

WHAT FILMS?

I settled with having people watch 3 or 4 two-minute scenes. The Criteria:

1. Must be something that the viewer will enjoy watching, and be riveted to for the entire two minutes.
2. Must be familiar and non-threatening (especially if it was going on Mechanical Turk)

So after a selection of around 20 movies, I narrowed them down to these four:

Pulp Fiction – Ezekiel 25:17 speech: Samuel L. Jackson carries this scene, and I wanted to see if the audience synced up when watching him.

Jurassic Park – The first brachiosaurus reveal – a scene with a clear high emotional point.

Apocalypse Now – The Helicopter Attack (Ride of the Valkyries) – Walter Murch edited this iconic scene.

The Little Mermaid – Kiss The Girl – a catchy, familiar tune that is hard not to like.

Movies that didn’t make the cut were: Carrie, the Prom Scene (too much?); Taxi Driver, “You Talkin’ To Me?”; Gene Kelly Singin’ In the Rain.

MECHANICAL TURK DOES NOT WANT YOU TO MAKE THE WORKERS DOWNLOAD ANYTHING

Was something that I found out after I had almost completed making a downloadable program, wasting a week.

Now that I think about it though, it may have been difficult to get workers to turn on their camera for a job that would pay them around $0.15 — I would have been paranoid about my privacy too, had I been them.

THE FILM DEPARTMENT WILL NOT REPLY TO YOUR REQUESTS TO PUT A BOOTH ON THEIR FLOOR WHEN IT IS FINALS TIME

No matter, I will try again in the Spring.

SO, IN THE END, I DID ALL THE WATCHING

I watched all 4 clips 4 times to see what I got. Given all the flaws of my software — motion detection instead of true blink detection — there was a pattern! There were certain areas, little clumps, where I would blink more often, and areas where I would not blink at all.

MOVIE POSTER

This is a picture of Walter Murch’s editing room. While editing, he lays out pictures of all the scenes on his wall so that he can, at a moment’s glance, glimpse the flow and content of the story.

I wanted to do the same thing to represent the blink data. When the viewer blinks, the corresponding frame is darkened. The more blinks happen within the area of the frame, the darker the frame is.

From the movie poster, you can see the areas where I have a tendency to blink by the darker areas (a frame where I blinked in the same spot every time are black!), and the areas where I keep my eyes open.

The results of the poster make me optimistic.

REAL TIME DATA

I wanted to show real-time blink data, so that you could see the precise moments that the blinks happened, as well as be able to compare your own blinks to the recorded ones while you were watching. There was no other way to represent this. I chose a row of ellipses that appear whenever I blinked (the X-coordinate is dictated by the session). I thought this was a simple and direct way to represent it for the meantime.

I was pleased with how the ellipses moved along the screen, visualizing the way my body reacted to the film as the film played.

The results themselves are revealing. For one, I noticed that I was blinking quite a lot in the data. The two possible reasons I can think of are that one, although I kept myself from moving, and even kept the movie screen size small to avoid this, perhaps the application logged small things like eye movements across the screen.

The more obvious reason for the noise is that because I wrote the program, and because I knew that I was logging blinks, I was constantly thinking about blinking the whole time I was watching the clips. And thinking about blinking is like thinking about yawning — the moment you think of it, you do it. You yawn, and I blinked.

As I continue to work on this, controlling this kind of noise and getting more accurate data is one of the more important but difficult challenges that I have to work on.

THE COMPONENTS AND THE CODE

Github repository here. Clicking the name of the application will take you to its main class.

Blink Recorder – The application I wrote to detect blinks. OpenCV with custom Haar Cascades, ProXML Libraries required.

Graph Generator – Simple graph plotter. The blinks are so dense that it does not really help, but it was a place to start.

Frame Saver – Takes a clip and exports frames for the poster generator.

Poster Generator – Generates the poster material (movie titles added in Photoshop).

Clip Generator – Live playback of the clips.

FINAL THOUGHTS

Even if I did not finish this project to the extent that I had planned, I am very pleased with the results. First of all, I learned a lot of programming, no matter how slowly I moved.

Secondly, tentatively speaking, it looks like Walter Murch’s theory is worth exploring further. It makes me excited to think about what the data will look like when there are hundreds of people watching the same clip. The idea of an entire audience syncing up when a film or piece of art is involving seems very beautiful to me.

This is a project that I have been thinking of for a while and would like to explore further, so I think I am going to pursue this for my thesis. There is a lot to work on:

1. Perfecting blink detection
2. What different situations? Not just movies?
3. How to represent this is other ways – perhaps sculpturally.

There was a suggestion during my finals presentation about creating a film that was edited by a collective of blinks — crowdsourcing the edit, in other words. This was one of the first ideas I had when I first got interested in this topic a few years ago. The more I thought about it, the less it sat well with me. I think the project should be about glimpsing moments when we as human beings synchronize in our train of thought, and the exploration and representation of that. Furthermore, editing is much more than knowing when to switch from one scene to another, or when to cut: it is about fitting together the correct pieces in order to tell a story. Saying that a good edit is when things are timed right based on the changing train of thought of a crowd would be oversimplifying it. A crowdsourced edit (I think) would not be either a good representation of when people synchronized, nor a good edit.

I have a little time to think about this further before the final semester starts. Updates will be forthcoming.

Happy New Year!

Im interested in using video to show data as a narrative. For a previous project, I had parsed text information and converted it as an XML for Final Cut Pro to import as video effects values. For this round I would like to affect the timing, cuts, and content, and not just video filters.

One simple example might be to show the progress or growth of something by using the progression of time. For instance, a shot of a child running or a woman singing could be playing forwards, and then backwards, and then forwards again depending on the growth of say, research on breast cancer over time.

In another scenario, and as a tribute to Lee Kuleshov’s experiment and film editing, I’d like to reorder the cuts within a sequence based on data. Ideally, when the process is repeated with the same material but different aspects of the same data set, one can demonstrate 1. a different perspective on the given data; 2. how film editing affects a story; 3. how the data changed the story.

Hopefully by next week I will have a better idea of what content I will need and what I would like to map to it.