Coding as Sketching

Zach Lieberman I have always loved motion. Even when I was an art student and studying printmaking, I was always thinking about animation and flip books and how you could take sequences of images and make something move. However, I didn’t go to film school and never knew how to do animation until I started programming. Then I was introduced to Flash. The thing about Flash is that you can write something like: this object position = X + 1 and then suddenly the simple line of text makes something move. For me, the essence of animation is really about bringing something to life. Some algorithms feel almost like living things. My job is to capture them in some way, be it with video or a high-resolution image. I feel almost like a wildlife photographer. I’m trying to find this really nice moment that conveys something about the system. Motion is really important because you can take the same graphical form and you can make it really happy. You can make it contemplative. You can make it subdued. You can make it extreme—just by adding movement.

ZL I really like this notion of simulation, where you’re setting up the system, and then you let the system evolve and see what happens. A common example is using particles and having them interact with each other. Then you add springs and forces and it’s like you’re building a sort of weird sculpture. Then, you see what happens: maybe it falls apart or maybe something interesting happens, but you don’t have keyframes and that makes it hard. Sometimes, when I work with clients who say, “ok it has to look like this,” I have to work backwards, say I will do things in reverse. I start with something that looks like what the client wants, and then I let my wild system evolve. Some sketches are hard to control, but the joy of being an artist is sometimes just tweaking, tweaking and tweaking this living thing and seeing what kind of magic happens.

ZL When I work with a client, it depends. Some clients have a really clear brief, sometimes they are very open. One of the things that’s really interesting about posting sketches [programmed experiments] on Instagram is that I’m getting clients to see what I do. So they start to see their work in my language and say, “OK, we would like that” or “we would like to see that, but with this change.” I did a project last year with WeTransfer, which was a kind of painting tool called Color Push. At the time, I was doing experiments with shaders and just trying to figure out how I could make colors flow, push out in different directions and feel very painterly . They were going through a brand redesign and they had their own idea about adding some graphical elements. We talked about building a sort of contemplative tool and I had this system. It resonated with them: it was very clear that we could use it and that it would fit in with their brand, but that really came out of sketching.

ZL One thing that I like about social media is you get a sense of where the mismatch is between things that I like and what others like. Sometimes I’ll make something that I like and people don’t like it that much, or I’ll make something that I don’t like that much, but people like a lot. Actually what I want is to understand. I want to understand myself better. I want to understand my intuitions. Part of it is trying to find a way to work publicly and openly, but that process is personal. So one of the things that I don’t do is ask for advice. People will say, “Oh, try this, try that,” but actually I want those ideas to come from me. I think it’s really important for artists and designers to develop their own intuition. In a lot of ways, sketching is like a diary to me.

ZL When I finished my undergraduate degree, it was the dot.com era. I got a job in a design studio and then the economy crashed. So I said, “I’ll go to graduate school,” because it was very scary in a way to be a student graduating. I got into a program at Parsons and one of my professors was Golan Levin, a former student of John Maeda’s. At the time I had been recreating John Maeda’s iconic posters in PostScript, like the Morisawa series.⁴4 John Maeda, Morisawa 10, 1996. “The Morisawa 10 series was inspired by a visit to a Shikō Munakata exhibition in Japan, and also by my fascination with emerging methods of working with letterforms on the computer. They were all composed in the Adobe PostScript language and were a little difficult to create because it was so expensive to make these images in the 90s.” John Maeda, http://b-o.fr/morisawa10 I showed it to Golan. He was very impressed, and, after I graduated, he invited me to work with him. At the time, we were using a sort of internal library of code that was from MIT but was not open source—openFrameworks was born from the desire to make an open source C++ library that I could use professionally, but also in the classroom.

If, in 2005, you had asked me how long I thought openFrameworks would be around for, I would not have said fifteen years. I’m not sure what the future holds for openFrameworks, but it definitely really opened up the possibilities for artists and designers that work with computer vision. The primary gift it gave people was to make them realize that you don’t need a computer science degree to use data from a camera and make things interact and work with OpenGL or those kinds of core technologies. I keep thinking about what the tools for the future might be: maybe they would be less monolithic and maybe they would be smaller and more lightweight or there would be more ways to stitch things together. If openFrameworks was the tool that democratized computer vision, what are the tools that democratize machine learning? What are the tools that democratize augmented reality, etc. I don’t know what those tools look like, but I’m excited about a healthy ecosystem. I’m really a big fan of p5.js, the JavaScript Processing community, because they’ve done a really great job of thinking about not only the tool, but also the ethics and the community.⁵5 p5.js is an open source JavaScript library for creating graphic and interactive experiences based on the core principles of Processing, https://p5js.org/ I don’t know what those future tools will look like, but I’m really excited about helping the next generation of people make them.

ZL The way modern visual coding systems like openFrameworks and Processing are set up is that they’re creative at the core. You start with a clear canvas and then, whatever you draw goes on top and on top and so on. So, the way you even think about it is very similar to how you might think of making a drawing or a painting, where you’re really starting with a background color, and then you add things. It feels like sketching to me, but it could be because of the term itself, I don’t know. It is the term used in Processing. I even think Processing maybe got that term from Design by Numbers. A lot of Processing’s DNA can be traced back to DBN.

ZL I am really interested in the body, particularly in how you can use it to understand computational systems. If you think about your fingers or your hand or the movement of your entire body, it’s so expressive, right? I understand the joy of playing a musical instrument as a very direct connection between what you do with your arms and your legs. This sort of energy you create physically has a direct feedback loop. Software is really exciting for me when it can be driven by motion and driven by the body. Sometimes I’ll try a graphical idea, and then I’ll say “Oh, what if I connected to a webcam?” Then you’re using your body to explore the parameter space rather than a slider, a mouse, or a keyboard. I always think about getting away from the computer screen. It is also a way to make things out in the world, things that are physical in one way or another.

ZL I went to art school. I studied fine art and printmaking and accidentally fell into the world of coding through design. I spent a lot of time reading design journals and books by authors like Steven Heller or Ellen Lupton. The work of Armin Hofmann was also very influential when I discovered coding. Then, what was really important to me was discovering John Maeda. I would go to the computer section of the bookstore. Every book was terrible: “Learn C++ in Ten Days” or “Learn Java in Twenty-One Days.” Then there was this book, Design by Numbers⁶6 John Maeda, Design By Numbers, (Cambridge, MA: MIT Press, 2001). by John Maeda. Design by Numbers was a book that presented the self-named crazy coding environment where you could write a “for” loop, but it was artificially slowed down so you could see and understand it. All the other computer books were about “how”: “How do you do this?” but, in DBN, Maeda was about “why”—he explained why you should do something. Why randomness, why iteration? Why think about the computer asset and designing machines? That approach opened doors for me. I still didn’t know that I could do this for a living, but for sure, Design by Numbers changed my life.

ZL Now I’m at the MIT Media Lab and one reason that I’m here is because those people and their students’ work mean so much to me. In a way, I feel sad that the tradition isn’t carried on. There have been other professors there, like David Small, who are really important, but design tradition is about ebb and flow. This is the first year I’m accepting students and we are asking ourselves, “How do we channel the spirit of the Aesthetics + Computation Group?” That’s really exciting. I teach a class called Recreating the Past: every week I talk about a different artist or designer, say, Vera Molnár, John Whitney, Muriel Cooper, Ken Knowlton, Lillian Schwartz, and so on. My students research them, take one work and recreate it using a modern tool. So they’re recoding, they’re almost reverse engineering, and then they’re coding again. I don’t teach classes that are design oriented, where you’re doing some kind of portfolio work or working on a commission. I teach classes that are more about having conversations with history.

ZL Of course I love moving images. When I see things like the Demo Festival and the way these screens are used in really creative ways, that makes me really excited. Still I am convinced that we really should be trying to apply the same levels of care and thought to the graphics that we make, be it on screen or on paper. Will these screens be treated with care? Will the images not be handled seriously because they are fleeting, because they just appear and disappear? There are probably some really beautiful things that can be done but screens have their own challenges compared to print media: just think about the flatness, the quality of projected light versus reflected light or the difference in quality of color. Currently, there’s a big controversy in New York about replacing printed maps, down in the subway, with digital information kiosks or screens. Obviously it’s not the same resolution, but you can show real-time information on a screen, you can have data visualization, you can zoom in, etc. I understand the sadness of losing the printed map, this kind of single image which was very high-resolution and had a quality to it.

ZL Just a couple of things about typography. I love the border of perception: when you take a word and you rotate it, or you stretch it, or you add noise, there’s a moment where it just doesn’t look like that word at all. If you apply enough transformation, there is this really interesting moment of seeing something and not seeing it, having it appear or having it disappear. I love playing with that edge because it’s about images that ask your brain to do a bit of work. There is a power in the type of images that come in through the eye and go to your brain and back to your eye and change how you see. Particularly with typography. The other thing about typography that I think is really interesting is that there are all of these esthetic decisions that go into making a typeface, right? If you look at the letter “a” with every font on your hard drive and see them one after another, you’ll see all of these tiny decisions that people have made: a little wider, a little thinner, a little tilted, etc. When I’m coding, it’s the same thing: the majority of what I do is adjusting, just changing numbers, changing numbers, over and over. There’s something about typography that really resonates with me, which is this sort of celebration of these tiny decisions that make something what it is. In my case, it’s trying to take math and turn it into some beautiful form.