SIM Sketches, Generative Art, UX/UI Research
Role: Artist, Interactive Designer, Researcher, Developer
Web Audio API, Anime js, React, Javascript, HTML5, CSS
Click images to experience the Sonic sketches in action.
As part of my ongoing research into how sound plays a role in immersive environments, micro-interaction, and audio cognition, I created a gallery of experimental works called “SIM Sketches” (Sound:Interaction:Motion). These sketches are used to investigate the intricacies of rich immersive interaction and sound-based user experiences. I use these studies to help inform my work in XR / VR / AR / AI / Digital Twin. Throughout digital and in immersive media, “Sound” often plays a secondary role helping to support the overall experience. In the past 6 years, Web-based audio api’s are becoming more and more advanced with examples like Meta XR Audio sdk and spacial audio, the rise of voice-based interfaces, low latency audio SDKs, SSML, to mention just a few. Also, UX Sound studios-like CMoore are helping to define sound UX style guides. To help me learn and be apart of this trend, I made this series of sketches to investigate and discover problems as well as solutions. To make these sketches, I am coding simple animated movements and using them as event targets for attaching audio using javascript, html5 canvas, and web audio API. The process has taught me about audio clipping, zero-point crossing, and best practices with audioParam. Currently, I am working to resolve problems with motion and audio crackle. Moving forward, I plan to explore sending audio through Web GL, GLSL shaders, and three.js. Here is a simple example.
 |  |
Our life experiences are rich with natural sounds, ambiance, distortion, harmonics, audio cues, and more. Visual design is fundamental in micro-interactions and user experience, but without sound, we are only given a fraction of the story. This research has potential applications in the areas of interaction design as well as embodied music cognition, generative music, and sound-based visualizations. This project helps me gain insight into how we can creatively use sound to contribute to an ever-expanding conversation on the user’s experience.
All this experimentation has led to a cool 3-minute short on display as part of the DownTown Houston+ Bayou Public Arts Project entitled “Bright Lights” which was projected on a 4-story high water wall. See Blog News for more
