[Research] TUIs in Sci-Fi Movies
Last edit: 4-Mar-2020
One of my favourite past times is watching movies. I mostly watch thrillers and actions with some Sci-Fi elements in them. In most cases these movies will involve a computer system with eye-catching graphics and sometimes fancy ways to use them. They are there for a reason: to promote the futuristic vibe of the movie world. And in order to make them more authentic, Sci-Fi movie creators are now working with interaction experts to design something that is backed with real research, instead of just randomly come up with something.
When talking about Sci-Fi movies creators working with HCI researchers, one recurring example is the Pre-Crime system in Minority Report (2003), which is used to investigate soon-to-be-committed crimes and prevent them from happening by catching the perpetrator(s) before they begin.
One of my favourite past times is watching movies. I mostly watch thrillers and actions with some Sci-Fi elements in them. In most cases these movies will involve a computer system with eye-catching graphics and sometimes fancy ways to use them. They are there for a reason: to promote the futuristic vibe of the movie world. And in order to make them more authentic, Sci-Fi movie creators are now working with interaction experts to design something that is backed with real research, instead of just randomly come up with something.
When talking about Sci-Fi movies creators working with HCI researchers, one recurring example is the Pre-Crime system in Minority Report (2003), which is used to investigate soon-to-be-committed crimes and prevent them from happening by catching the perpetrator(s) before they begin.
To create this system, Steven Spielberg (director of the movie) and his production team worked with Jeff Underkoffler, a grad student from MIT, and his lab to answer the following question: What will computers look like in 50 years? [A] What they came up with has inspired many projects and research projects using gestures as the input language, also known as Gestural Interfaces.
However, in this research project I decided to look into something slightly less talked about.
If you pay closer attention to the lower left corner of the image above, you'll see two wooden ball. What are they, you ask? Well, they are tangible representations of the victims and perpetrators of a soon-to-be-committed crime.
However, in this research project I decided to look into something slightly less talked about.
If you pay closer attention to the lower left corner of the image above, you'll see two wooden ball. What are they, you ask? Well, they are tangible representations of the victims and perpetrators of a soon-to-be-committed crime.
When a crime is detected, the system gives alerts about it using two colour-coded, uniquely-grained wooden balls—one engraved with the perpetrator(s)’ name, the other engraved with the victim(s)’. These balls are later on placed in a cradle to identify the persons involved.
The idea of representing events or people using a physical ball has been proposed before, by Bishop in his Marble Answering Machine, which is crowned as one of the first examples of Tangible User Interfaces (TUIs), and this has made me ask the following two questions: How do tangible systems depicted in Sci-Fi movies differ from those conceptualized in the TUI research community? And what do these Sci-Fi TUIs tell us about the future of tangible computing?
With these questions in mind I worked with my supervisor, Dr. Alissa Antle, to systematically analyze tangible systems that appear in Sci-Fi movies.
The analysis
We started off by looking for Sci-Fi movies that have tangible systems in them. Our selection criteria for the example systems are: (1) at least one tangible object is directly used in the system, (2) the system exists in the future or a settlement unlike our current state of reality (e.g., a hidden organization with advanced technology), and (3) the system is used computationally for purposes advancing the narrative of the movie. We chose these movie examples because they are relatively well-known, thus likely familiar to our audience, and there is similar prior art in the TUI literature. Also, our selection cover a variety of forms, contexts, and technologies, comparable with current research.
The idea of representing events or people using a physical ball has been proposed before, by Bishop in his Marble Answering Machine, which is crowned as one of the first examples of Tangible User Interfaces (TUIs), and this has made me ask the following two questions: How do tangible systems depicted in Sci-Fi movies differ from those conceptualized in the TUI research community? And what do these Sci-Fi TUIs tell us about the future of tangible computing?
With these questions in mind I worked with my supervisor, Dr. Alissa Antle, to systematically analyze tangible systems that appear in Sci-Fi movies.
The analysis
We started off by looking for Sci-Fi movies that have tangible systems in them. Our selection criteria for the example systems are: (1) at least one tangible object is directly used in the system, (2) the system exists in the future or a settlement unlike our current state of reality (e.g., a hidden organization with advanced technology), and (3) the system is used computationally for purposes advancing the narrative of the movie. We chose these movie examples because they are relatively well-known, thus likely familiar to our audience, and there is similar prior art in the TUI literature. Also, our selection cover a variety of forms, contexts, and technologies, comparable with current research.
We have selected Minority Report (2003), The Island (2005), and Black Panther (2018). We also included an "anti-example", Iron Man (2008), featuring an interface that is popularized by many recent Sci-Fi movies yet deviates from TUIs in one key aspect: lack of physical objects in the interaction. We believe this example could demonstrate the myriad possibility of futuristic systems and inspire the future of TUIs.
To better connect the movies to the TUI literature, we use the MCRpd model proposed by Ullmer & Ishii (2000) [B] to analyze the systems in each of the movies we picked.
Findings
I would really encourage you to read the paper to get the full details of the findings. But long story short, we have two main findings: (1) Sci-Fi TUIs do not follow the MCRpd model completely, and (2) Sci-Fi TUIs are tailored for rapid viewership.
What Finding#1 means is that there are similarities and differences between Sci-Fi and real-world systems. We believe that the similarities can tell us how TUI concepts are manifested in Sci-Fi movies, and the differences can tell us how Sci-Fi movies adopt other interaction types.
What Finding#2 means is that Sci-Fi systems are made in a way so that the audience can understand how they work in a very short period of time. Real-world systems can learn from that by (1) using visual and audio cues to make them more engaging, (2) combining other interaction paradigms to expand interactivity, (3) using smaller number of visually capturing tangible objects for better affordance, and (4) using context to design a more coherent look and feel.
One more thing we found was that the design of Sci-Fi systems is similar to speculative & fictional design practices. We believe TUIs from the research community can allow movie creators and fictional designers to design better fictional systems, by using TUI concepts to design things that appear effective, socially-situated, and familiar to the often-passive audience.
We presented this work at the TEI conference in 2020, and won the People's Choice Best Talk Award.
What's next
I really like the idea of looking at Sci-Fi worlds to get insights about how our future computers will look like. But in a grander scheme what I am doing is to investigate impacts of popular culture on technologies and design of computer interfaces. I intent to pursue this direction with interaction paradigms other than tangibles.
So yeah, watch more movies and do more research!
To better connect the movies to the TUI literature, we use the MCRpd model proposed by Ullmer & Ishii (2000) [B] to analyze the systems in each of the movies we picked.
Findings
I would really encourage you to read the paper to get the full details of the findings. But long story short, we have two main findings: (1) Sci-Fi TUIs do not follow the MCRpd model completely, and (2) Sci-Fi TUIs are tailored for rapid viewership.
What Finding#1 means is that there are similarities and differences between Sci-Fi and real-world systems. We believe that the similarities can tell us how TUI concepts are manifested in Sci-Fi movies, and the differences can tell us how Sci-Fi movies adopt other interaction types.
What Finding#2 means is that Sci-Fi systems are made in a way so that the audience can understand how they work in a very short period of time. Real-world systems can learn from that by (1) using visual and audio cues to make them more engaging, (2) combining other interaction paradigms to expand interactivity, (3) using smaller number of visually capturing tangible objects for better affordance, and (4) using context to design a more coherent look and feel.
One more thing we found was that the design of Sci-Fi systems is similar to speculative & fictional design practices. We believe TUIs from the research community can allow movie creators and fictional designers to design better fictional systems, by using TUI concepts to design things that appear effective, socially-situated, and familiar to the often-passive audience.
We presented this work at the TEI conference in 2020, and won the People's Choice Best Talk Award.
What's next
I really like the idea of looking at Sci-Fi worlds to get insights about how our future computers will look like. But in a grander scheme what I am doing is to investigate impacts of popular culture on technologies and design of computer interfaces. I intent to pursue this direction with interaction paradigms other than tangibles.
So yeah, watch more movies and do more research!
Related publication:
[1] Cheung, V. and Antle, A.N. (2020). Tangible Interfaces and Interactions in Sci-Fi Movies: A Glimpse at the Possible Future of TUIs through Fictional Tangible Systems. In Proc. of TEI 2020: ACM Conference on Tangible, Embedded, and Embodied Interaction. February 9-12, Sydney, NSW, Australia. p393-401.
https://dl.acm.org/doi/abs/10.1145/3374920.3374942
** People’s Choice Best Talk Award **
References:
[A] Dave Smith and Corey Protin 2015. The visionary behind the mindblowing technology of ‘Minority Report’’ has built it for real.’
[B] Ullmer, B. and Ishii, H. 2000. Emerging Frameworks for Tangible User Interfaces. IBM Syst. J. 39, 3–4 (Jul. 2000), 915–931. DOI:https://doi.org/10.1147/sj.393.0915.
[1] Cheung, V. and Antle, A.N. (2020). Tangible Interfaces and Interactions in Sci-Fi Movies: A Glimpse at the Possible Future of TUIs through Fictional Tangible Systems. In Proc. of TEI 2020: ACM Conference on Tangible, Embedded, and Embodied Interaction. February 9-12, Sydney, NSW, Australia. p393-401.
https://dl.acm.org/doi/abs/10.1145/3374920.3374942
** People’s Choice Best Talk Award **
References:
[A] Dave Smith and Corey Protin 2015. The visionary behind the mindblowing technology of ‘Minority Report’’ has built it for real.’
[B] Ullmer, B. and Ishii, H. 2000. Emerging Frameworks for Tangible User Interfaces. IBM Syst. J. 39, 3–4 (Jul. 2000), 915–931. DOI:https://doi.org/10.1147/sj.393.0915.