ICM Final

Concept 1: VJ experience
The very earliest concept is to create an self-enjoyed environment in a 3D space. This immersive environment has video fed in behind which creates a VJ experience… and connected with VR devices..


Concept 2: Identities & Surveillance
Gazing others seem to be a normal behavior; however, most people feel uncomfortable when they are being stared too long. What if you are being stared without noticing? This is causing people feel insecure.  you would not know where do people’s gaze at when they are wearing sunglasses. You do not know if you are being watched. We thought if we have multiple players connected through different webcam and feed all of the webcam randomly to all the spheres.

While brainstorming….

Have the WebRTC worked successfully on both ends…


Concept 3: Uncertainty/People watching others and being watched and lost in the reality
.. So in order to simulate this situation….we came up with a chat room that the users think they are chatting with the others in live, but those are pre-render videos. To set up a network issues would foster the environment becomes more real and in the current time.


After user testing… we got so many feedback…
*pick a better background which does not tell the location and time
*if the actor and actress were scripted might be more real
*should invite the user into the conversations


Final Version: We took some of the advices that in order to be more “real,” we should have a start button before the user enter into the chatroom. So, we created a disclaimer page for the user before they began..one reason was because while user testing, some people took it seriously when they knew the whole experience was fake…so we tried to have some sort of notice before the user begins. Also, this will allow them to begin the experiment. Later, the chat began…and after 40 seconds, the user’s recording would shift to the left one and replace its previous video. Then, there will be an alert asking the user if they want to download it and tricked the next user. If they clicked yes, files would be downloaded to their local folder, if not, the experiment will start over and go back to the disclaimer page. So, each round, the recording will be saved and feed into the next screen from left to right. Eventually, new user will be tricked by the previous users and got confused.


Code Setup
This repo sets up and runs a fake video chat site that loops webcam-feeds of users. The aim is to explore the nature of virtual video communications: How do we establish communications online with strangers that we see and hear – but do not know whether they are real or not? How is trust established in a virtual group experience? How do we perceive our role in a real (or fake) group? How do we feel about unreal experiences? What makes us willing to participate in fake schemes?

run python3 -m http-server in repo folder
edit move_files.py, replace folder names with corresponding local folder structure
run move_files.py in background to move around recorded video bits (use sudo if needed)

The start page is index.html that asks users to agree with terms of use. The button at the bottom is linked to vid_chat2.html – which sets up the virtual chatroom with main_vid_nu.js. In main_vid_nu.js we use webRTC to record a stream from the user’s webcam. This is positioned in the middle of the screen, the two side screens play back two pre-recorded videos (maxresize.mp4 and resizeBav.mp4). Those videos display the (recorded) webcamstream from two actors. They seem to try to establish a communication between each other. These videos are used only in the first two rounds. After that two streams from the webcam have been recorded and get fed into the left and right screen. This “replacement” is repeated in every round, the videochat is looped with user-videos. Before this takes place, the user will see a recording of his own webcam from the last 40 seconds popping up on the left screen. After a few seconds a popup-alert asks the user if she wants to fool other people with her own recording. If the user agrees, the video gets downloaded as a video file with a specific name (repl_left or repl_right). The websites defaults back to the home screen index.html file. The move_files.py script continuously waits for videofiles to come into the downloads folder. If a new file comes in, it moves the file to the main folder and stores the name of the last downloaded file into move_vids.txt (located in the main folder). This enables a form of tracking of which side of the screen (right or left screen) should be updated with the last video – right and left side of the vid chat are always fed the last recording from the last user in an alternating pattern. Python is necessary to circumvent the security restrictions in JS.

This is probably a very complicated way to run such a chat app locally. It is used here to quickly prototype the idea of a fake video-chat that loops in itself.

recording from webRCT – code based on https://github.com/webrtc/samples/tree/gh-pages/src/content/getusermedia/record
video screen – code based on https://github.com/mrdoob/three.js/blob/dev/examples/canvas_materials_video.html

Final Proposal_Immersive Livestream Concert with VR Gear

Last week, I made a cube that was 3-dimensional which constructed with multiple boxes rotating. Each side of the box is built with camera capture so that the person is see himself/herself inside a 3-D world in 360 degree. Also, the user is able drag the mouse to see the 360 degree vision of the space. In the back of the cube, there’s video playing which allow users to have a thrilling vj/sound experience in 3-D world.

Previous Project…

Improve the surface and the environment. Also, one user is inside the VR world, they can interact with the content.

Support sources…

I found an library that allows javascript sketches to run in unity, which is called A-Frame and I found an example that is very similar to my idea, click


ICM week 8: Immersive vj experience

Thoughts & Ideas:
I want to create an immersive experience for user to have their own world to enjoy the music and visual. It’s mimicking the concept of VR world which allow the user to virtually participating the event inside the screen by using camera to make the event present in live.

However, I encountered the problem of installing the video/GIF to the background. Then, I had to first move the center of the camera outside the cube. Then, I will be able to calculate the distance away from the cube, which will be visible. If you thinking 3 dimensional, by going further away from the cube is actually gradually moving negative in z axis.  After understand the concept, I translated the position of the video to negative so it became the background.

While adjusting the position of the video, I realized “translate” is actually accumulated. Since I translate my center in draw function, my sketch is kept looping and that might be the reason why translate is being accumulated?? Also, I have two different shapes in this sketch, box and torus. Right now, they both are reflecting the images from live camera captures; however, that’s not what I want. If you look into my code, you will see that only the boxes are in texture of camera captures but not toruses. I wonder this might share the same problems with my previous challenges but I tried to have different texture for toruses, somehow did not make any change.

Demonstration Video:
Immersive World Version1

Final Version:

P5 version: Click

var video;
var bg;

function setup(){
createCanvas(800, 600, WEBGL);
video = createCapture(VIDEO);
bg = createVideo(“beats.mp4”);

function draw (){
// video(0,0);


ambientLight(0, 0, 10, 3, 3, .8);
pointLight(300, 300, 300, 300, 300, 1);
// image(bg,0,0,width,height);


var radius = width * 1.5;


// translate(0, 0, -600);
for (let i = 0; i <=12; i++){
for(var e = 0; e<=12; e++){
var a = i/12 * PI;
var b = e/12 * PI;
translate(sin(2 * a) * radius * sin(b), cos(b) * radius / 2 , cos(2 * a) * radius * sin(b));
if(e%2 === 0){
// fill(80,20,64,64);
rotateZ(frameCount * 0.01);
rotateX(frameCount * 0.01);
rotateY(frameCount * 0.01);

ICM Week 5 & 6: Array / Classes / Object / HTML Slider

For this week, I wanted to combine my PCOM and ICM together; so…I used a pulse sensor to adjust the speed of the heart…but..the result wasn’t significant..At the end, I did not use pulse sensor but instead of using the potentiometer..

First Version (Codes)

Final Version (Codes)

I still couldn’t figure out how to make my ellipses look more like the actual fog..(maybe to blur out the edges of the shapes?)

Week 4: function

Function of flowers (Codes)
By using the function, I created five flowers..

Petals rotate around the center ellipse (Codes)
I started to test on one flower first…

Break it down…Function of map (Codes)
Before I drew the sun and the sky, I thought I would need help to make it happen. However, I decided to try on myself and succeeded! I was very proud of myself and felt super motivated to learn codes even though the codes might not be that hard for most of the people. BUT, I solve the problem all by myself!

Version 1 (Codes)
Finally, I tried to make the cat’s tail to move with the mouse.

Version 2 (Codes)
This version creates a optical illusion according to the speed of the rotating petals. By changing the speed of each flower in draw function, all the flowers would be affected by the largest number…which is the problem I would like to be solved…

I created a lot of functions which also make a lot of objects relevant to each other. So, I couldn’t figure out what would be the easiest way to change the speed of each flower and the height of each stick on flower….

Week 3: Group Project

Me, Isa, and Camilla came up with the idea of design a perspective room with views outside the window. First, Isa came up the design and codes of the slider which we thought we could turn that into the curtain. When the curtain is turn on, we will be able to see the rain outside the window. Then, I drew a character, No Face from the animation Spirit Away, inside the room. Originally, we wanted the character to move around the room. When he leaves the room, the light will turn off. However, that was too complicate it. So, we simplified it.

Pieces from each of us…


Click for codes

Problem we have faced…
Doing our own work was not difficult; however, combine all of our work together was hard. We first encountered into the problem of putting the rain inside the window. We tried several times before we ran to the resident.

Progress of the rain…
Click for codes

Yuli helped us to solve the problem which helped! She used the timer function which I am still a little confused… yay! Click for Codes

Combine the parts…
We first combined the rain and the room together, which is what it looks like in below.. Click for codes

Progress of the light…
Then, we wanted to make the room super bright when we click the light on top of the ceiling, which is similar to the function in below…

Progress of the slider…
And, we tried to put the slider (shades) in front of the window/rain, so that we can have our window seems more completed.

Install button function…
However, there are some weird illusions or design appear…but I actually like it.. Click for codes

Final Version…
Then, I asked my friend to help us to furnish the work. So, by pushing the button, the room will turn into bright yellow and the shades will cover the window. Click for codes

Week 2

Sketch & Initial Thoughts…
I had a hard time to figure out what I want to draw. So, I began to search for ideas by looking at other people’s work then to develop on top of it. Without giving the meaning on my initial sketches, I just began drawing some geometric shapes. Then, I found out that these actually look like a radar, a fan, a lighthouse…etc.

Process While Working…
During my production time, I seek out lots of help from many experts and learnt from the experience. I realize this is the best way for me to learn the techniques. I began to draw the ellipses and the quadrate. To make it more fun, I brought in the rotation to the ellipses and the mouse tracked function to the quadrate. By switching the origin to the center of the canvas, both quadrate and ellipses will spin around the center point of the canvas. However, the quadrate follows the mouse and rotate.

Then, I started working on the color. I make the colors change randomly on both the quadrates and the utter ellipse. Also, by clicking the canvas, the background color will change randomly as well.

Later, I made another version, which I exchanged the effects on both the quadrate and the ellipses. So, the quadrate spins clockwise; instead, the ellipses follow my mouse to rotate. But, the color effects remain the same.


Problems that I’ve encountered…
I still can not figure out how to change the speed on color, both the quadrate and the ellipse. I tried to implement a fish in the canvas that will bounce back and forth in the canvas.


Version 1


Version 2

Computation Applies to My Interests…

I began to seeing computation is necessary since I get in touch with mobile application design, interactive installations, and creative codings on some websites. Since my background is graphic design, computations can totally apply on UI/UX design which allows me to go beyond my career. By learning computations and creative coding, I will be able to develop ideas not just on the screen, but also physical objects with the knowledge.

Recently, I fall in love with interactive installations. They are not limited to the installations in the museums and exhibitions, but also those products that actually solve people’s daily problems. For example, smart mirror that are installed in some department store. Smart mirror is able to give suggestions to the customers which solve the problems of deciding and picking. So, I think is so much fun to incorporate technologies and creative ideas to enhance our daily life. We may not completely solve the problems but we are able to make our life easier or more enjoyable.

Week 1: About me

Initial Sketch
Instead of depicting my portrait, I go with showing my inner world. Originally, I was planning to use colors of blue, yellow with white canvas. Later, I realize that blue strokes is very eye-catching which is not the effect that I want to present.


Final Look
On the canvas, there are several ellipses and arcs. The arcs surrounding the ellipses represent the public and the world. The ellipses represent my inner world, which separate in several layers. The outermost ellipse has the thickest stroke weight but the softest, meaning that I am friendly, soft but take time to become close friend in the beginning. From the outermost ellipse to the innermost one, the stroke weights decrease but stroke colors become stronger. Finally, there is a diamond that contains two triangles at the center. The triangles represent the most secret part of me that I hardly share the space with others. Therefore, the shape of diamond with no filled is the last protection of my most mysterious part. The lines inside the ellipses represent the guardians to protect my spirit.

Orange is a vivid, vital, and energetic color that is similar to NYC, a city that never sleeps.  Yellow represent myself which is bright, optimistic, and cheerful. Also, by using the transparency, the outermost stroke of the ellipse smoothly blend in with the background color that shows I’m able to communicate and blend into the society.

Codes on p5. js

Problems that I’ve encountered
One of the difficulty while working on p5. js was calculating the parameters, especially arcs and quadrilaterals. I am still unclear about the angle to start the arc and mode. Other than that, I just had to be patient and calculate, try, and test the numbers to get the best results.