YRANOITCIP_final project

This project is originated from the idea of Pictionary, a game where one player draws the picture of something on a board while other teammates guess what it is. It is always funny to see all kinds of guesses the teammates come up with. After this class about text generation, the idea of an inverse Pictionary game occurred to me. Can we design an inverse game of Pictionary, where computer generates the text, i.e. instructions, and players try to draw it?

We need to give the computer some basic “knowledge” about how drawing instructions should look like. It would be ideal if we could have a huge “drawing instructions” dataset so we could use a RNN model to train the computer. However, such dataset is hard to obtain. Furthermore, since we need to make sure that the instructions make sense to the game participants based on their current situation, I developed a minimal template based on my expected scenario. The computer should be able to “improvise” based on the current situation, so do the participants. The computer arbitrarily changes certain parameters in the template in order to generate different texts. I hardcoded some of the parameters in order to cope with the environment. In order to be more creative, I use Spacy to let the computer find similar words in the dictionary for other parameters.

Another level of flexibility of the text generator roots on the interaction between the generated text and the environment. The instructions take in elements from the current participants and audiences and blend them into the drawings. So the final product would be a mixture of the hardcoded parameters, creativity of Spacy similarity semantics, and randomness taken from the current environments.


The following screenshots are two versions of generated drawing instructions:




And the source code:

One Universe at Different Times_Progress Update

Our inspiration:

  1. Einstein’s Dream:
    Multiple Universes Parallel / Intersect at Different Time

  2. Multiple Intersecting Universes:
    create multiple layers to represent multiple universes within each other different levels of layer with different transparency..
    inspired by Takahiro Kurashima

Working Progress:

  1. One Universe at Different Times
    Working with three.js

2. Combine ideas with zoetrop
Transfer to A-Frame

Next Step: Build another storyline from another perspective:

  1. Peak behind one of the face & discover something new
  2. Go forward to one of the face & the face react back to user
  3. Look at the face at different angles – user discovers its nuances, familiar yet different

Week 4_vision of future storytelling

How human communications needs will combine with emerging technologies to change the media landscape in the near future?

synchronization in time and space
Human communication takes forms of either person-to-person individual information exchange, including chatting or giving a talking to, or broadcasting-like one-to-all information conveying and sharing, such as mass media and storytelling. Both forms require two types of vicinity: temporal synchronization and spatial locality.

temporal synchronization
The first category of vicinity, temporal synchronization has been widely addressed throughout the history of technological development: besides talking face to face, the mailing system is the most primitive technology the human kind uses to communicate with others. Development of this technology experienced the period of using human power, animals (including horses and pigeons), to machineries (such as cars and planes). Those technologies largely reduce the delay in temporal synchronization among human communication. Sophisticated mailing systems makes it possible to handle large communication throughput, such as the well designed logistic algorithms and protocols. On the other side of the horizon, broadcasting media-based communication or storytelling starts with the spread of rumors, hanging of posters, printing of newspapers. Those technologies of communication are also largely dependent on means of transportations such as delivery by human, car, or plane. With the approach of Electric Age, both forms of communication got accelerated to further satisfy our need for temporal synchronization: telephones firstly made live communication over large distances possible. It started with landline phones and developed to cell phones, shortened the communication delay from spending the time to reach a land line to picking up a phone right away. Telegraph makes media storytelling faster as well. News and stories happening all over the global could be broadcast by radio wave and shared with people on the other side of the world via printing media. The advent of information age completely eliminated the communication delay and enables absolute temporal synchronization. People use FaceTime to talk to people on the fly. Internet websites can publish and share events and stories at the first second. Medias such as Instagram and Facebook even makes individual live storytelling possible by all the live video sharing functionalities. Technologies have confidently satisfied our communication needs on temporal synchronization.

On the other hand, the spatial locality has not been well tackled, despite our ardent desire for it. With telephone, people are far from being located in the same space when communicating with each other. The invention of camera gives mass media a chance to sort of bring the space where the event or story happened into the audience by a 2 dimensional projection of the space printed on a plane: photographs. Video recorders and TVs added another dimension to the projection: time, which is, however, far from realistic. People also uses sound to improve the lower-dimensional replica, in order to approximate spatial locality. An emerging technology, Virtual Reality, is one significant step further to realize true spatial locality. The 360-degree view created by VR plus head and gaze tracking has brought our pursuit for spatial locality during communication into a whole new level: people can talk to their friends using VR, just as if the friend were sitting next to him/her. Media could broadcast breaking news of stories by project the whole scene to their audiences via VR. These could be easily implemented in the near future by this emerging technology.

Although the current VR technologies allow the deception of our eyes and ears, other senses could also be satisfied potentially by the next level of VR: haptic, smell, taste, etc. The satisfaction of these senses will further increase the fidelity of our simulated spatial locality. These are the next level of Virtual Reality we could use in our communication of either forms, and a pseudo-trueness of the temporal and spatial vicinity could be achieved at the same time. However, pseudo is after all pseudo. The boldest vision of human kind to achieve spatial locality during human communication with absolute fidelity is teleportation. Only with means of teleportation can people communicate with each other over large distances within the same space. We could talk to a friend after we are teleported to them. Media could tell their stories by teleport their audiences to the space where the story is actually happening. Teleportation is far from emerging technologies, but one of the drives behind the research toward it would also be our communication needs for spatial locality.


This week, me and Alan began the project by scanning 3D models of ourselves. And, we took a video on both of us of sneezing. Later, we used slow motion to capture each frame and convert each motion into a panoramas.

Right now, we successfully import the panoramas with my 3D scanned object into the environment.


Next, we tried to connect the environment with Kinect and Kinectron. The code seems to grab the data from Kinect successfully; however, there’s nothing happen….


Assignment 6_drawing instruction

So, I thought of doing a live computer generated drawing instructions using spaCy’s built-in word vectors and annoy vectors semantic similarity for this week’s assignment. The drawing will be produced base on the current environment. I’m thinking to develop this project into my final. To improve, I want to make the script to be more complicated and be more interesting so that the drawing will be more arbitrary and the instruction to be more specific.


Week 3_ideas of the storylines

For this week, me and Alan had conversations about the ideas of how to intersect multiple stories in different spaces. We were inspired by the book, Einstein’s Dream, about the spaces that exist in different time. How can we bridge multiple time zones together and tell a story? This is the discussion that we have been discussing over the week. We decide to start brainstorming for final project and break down our working process starting from this week.

And, we came up with some ideas:

  1. Two Spaces Intersecting in Different Time
    We were thinking to shoot a 360 video at a public place, such as a coffee shot. The video captures the space at a fixed angle for couple hours. And, bring in a 3D model while working in three.js and set a focal point on a chair and table. The idea is just create a spot light for this location and bring audience attention of what to see and also show the time at the coffee shop. We were thinking even layering another video footage of a person interacting with the video that we shot at the coffee shop. So, a person sitting on a chair (laying with a 3D model chair) and the person from another video is placed on another chair that at the same table working independently.
  2. Two Spaces Laying on Top of Each Other in Different Time
    To extend the idea from the previous one; instead, we change the composition of the videos. For example, a second video is placed outside the first video. The camera is placing at the center of the two videos so when the inner video have lower transparency, the audience can see the outer video (second one).
  3. Alan Sneezing
    So, we thought if we 3D scan Alan’s head and duplicates to min 4 heads and place them into a circle all facing toward the center. Each head’s facial expression captures Alan’s sneezing moment starting from the beginning to after sneezing. And, when the camera turns to the last head (when the sneeze happens), something pop up and throw to the camera.


  1. Bring memory back to life
    With the increasing number of elderly which suffer from Alzheimer’s, technologies that could improve their quality of life and probably alleviate their symptom would be desirable. If we can implant the mindset of the patient’s close family or friend into a 3D printed human body of that person, and let him/her keep communicate with the patient, this can bring some of the memory back and probably immerse the patient in the joyful moments in the past.
  2. Swipe Left or Swipe Right
    Dating apps are becoming more and more popular with the increasingly pace of life. The so called “scientific proven” matching algorithms behind those apps are always under question. We can somehow develop a personality emulator that creates our virtual embodiment and evolve in a computer program to simulate all the intricacies happened in inter-personal relationships and record details involved in dating couples and rate them based on certain metrics. This would be a more “scientific proven” matching method.

  3. Spiritual Embodiment
    Is our spirit defined by our physical embodiment or does the latter shape/appearance become the former? If humans would have the body of a dog, would our common mindset be totally different from the one we currently possess? If a dog owns the spirit and consciousness of a human being, would their dog-like body evolve to cater the superiority of a human mind, probably capable of intellectual endeavors such as dancing, coding, deception, or empathy?

Project idea of using Syphon and Isadora or other softwares:
Medium: Syphon + Isadora + processing + perception neurons live motion capture
Project name: VJ Manipulation and/or Variation

The idea originates from Karlovy Lázně, the biggest music club in central Europe, which contains five different floors, with each floor playing a different style of music and visual. However, as one music club, one missing piece is the union of all the artistic components. The idea is to originate visuals of the five different floors based on one common raw video stream, created by either a common base music or a main DJ’s motion. Through Syphon the base visual could be shared and streamed to different Isadora programs to manipulate the base visual based on each floor’s different theme/style. In this case, the different themes or styles of each floor is maintained while the unifying theme as one single biggest music club in central Europe is also introduced, since all the manipulated VJs through Isadora are implemented through Syphon sharing from a common based visual.


MOMA visit

This MOMA trip was actually more inspiring than I thought. Although I have been to MOMA for so many times and seeing those painting so many times, I never had a chance to have someone stand next to me and explain its process.

We visited several paintings but there were few ones that caught my attention.

  1. Reflections of Clouds on the Water-Lily Pond by Monet:
    The presentation that MOMA presents is interesting. When I first walked into the space, my eye were invited to this three canvases and this immersive experience makes me feel like I’m inside the painting. Although this painting is broken down into three canvases; however, this concave setting is bring all as one that creates an interesting on point perspective.



2. Louise Rosenblatt Literature as Exploration: 
According to Katie’s explanation, this painting was painting during the war and is categorized under Surrealism. The perspective of this painting is quite interesting because there is sort of a layer or depth inside of this painting. There’s a painting inside a painting which is depicted that the present world is more complicated than we actually imagine. Also, seeing the wireframe of the building can be a metaphor of seeing the truth behind.


3. Shovel by Marcel Duchamp:
First when I see this as an art piece, I was like “why?”. But after hearing the story behind, I understand the reason and was actually quite impressed and thankful by him. So, being the pioneer of modernism, he submitted this shovel to an art competition and called this as art when everyone else sent in their beautiful paintings.















4. 3 standard stoppages by Marcel Duchamp:
System: Duchamp made fun of the system that our society comes up with. The reason why we have system today is because our brain can only process so much of information and that’s why we have system. The process he came up with this was also pretty fun. So, Duchamp dropped the meter randomly on the floor and found out the patterns. Then, he made his own system to challenge the systems that existed in our society.

Honestly, after viewing several modern artworks, I really think the most valuable part is not their artwork, but their concept and message behind. So, I think the best way to present these modern artwork is to include their production process with solid description of their ideas while creating the art. I think this is better than just showing the art itself.

5. Painting (don’t quite remember the full title and artist’s name)
This painting is not as well known or recognized compared to those previous ones. However, the presentation of this painting is interesting. The layers of paints somehow creates a 3-Dimensional space and turns the painting becomes more alive. Also, since the paints are layering up, it creates the digital form.





Einstein Dreams_interlude 1 through “29 May 1905”

I read the part from interlude 1 through 29 May 1905 with the topic of discovering time. In the beginning, I found no relationship between each chapter; however, I realized the author continues to discover and discuss from different perspectives, which I found really interesting and inspiring.

The chapter 8 May 1905 is talking about the world will end soon at a specific day in the future and everyone remains the same time to be alive, which the world becomes peaceful without crime or negative feelings. This immediately reminds me the idea of Socialism. Things are distributed equally to everyone in order to make everyone equal. However, this is still not a perfect solutions to make everyone happy with what they have. Also, people can suffer together but hard to share with others because greed has no limits. Today, people are trying hard to extend life and trying to live as long as possible even they are lying on bed in the hospital. But, there’s no happiness anymore; instead, with a limited of time, people are actually happier and enjoy the most of their life.

This also refers to another chapter, 14 May 1905, which talks about without time there’s no life. Some disagree but I do agree. With a fixed and frozen of life, is like extracting the best part of your life and store as memory to others and die with happiness.

The next chapters, 15 May 1905 and 20 May 1905, talk about memory. After reading these two chapters, it brings up a thought that if in the future should people have the ability to delete their memories or not. With memories, our lives become richer; without, we become like a robot. A lot of emotion will be lost; however, people might be more efficient, easier to be friends with, and even become better human being.

Assignment 4_Tracery+Markov Chain

This week, I decided to use the Sonnet file that I created with Tracery couple weeks ago. I wanted to test it if Markov Chain’s model can duplicate similar text generation behavior.

So, using Tracery library, I used artificially designed template to generate Sonnet. The rules of text generation were explicitly articulated in the form of a template. Thanks to the library, I can replace some text components in the sentence.

On the other hand, I want to explore the possibilities of using Markov Chain to achieve the same behavior. Based on a relatively large Sonnet text data set, which was generated by Tracery library, a Markov model was built. It captures the interrelationship between individual and multiple characters and words, using the concept of n-grams. I used a similar function definition from the class notes and the Markov model is essentially a dictionary or look up table. Different keys in the dictionary correspond to multiple text predictions. The text generation was based on the same idea as in the class notes.

After the experiment, my Markov model is able to automatically learn the text generation rules previous specified by an artificial template. At the sentence level, the Markov model achieves the same effect as Tracery’s template. The grammar and syntax remain intact. However, Markov model picks random starting point and the whole format of Sonnet cannot be guaranteed.