This years’ NIME conference, which took place at Goldsmiths University of London, featured a one day hackathon centered around assistive interfaces narrowing down disabling barriers to musical and creative expression. The challenge was the following: “Design a musical device that can be easily configured/modified to make music making accessible to all, including musicians with a disability”.
A hack by a member of our research team, Fiore Martin, won the prize related to this challenge!
Visual programming languages such as Max/Msp and Pure Data (PD) are well established in the community of sound designers, interactive artists, creative coders etc. However they are exclusively based on visual diagrams – small shapes representing unit generators, messages, numbers etc. and connection lines describing the signal flow through such objects – and are therefore inherently inaccessible to visually impaired users.
The CCmI Editor, which is based on the work I developed through my PhD research, sports a range of default diagram types (UML, tube map, mind maps, flow charts, circuit diagrams, etc.), but can be easily customized to accommodate new user-created types through its template interface and even more complex types by coding extensions of the nodes and edges of a diagram.
Hence the idea for this hackathon: extend the CCmI editor to make it mirror the PD language and hack it in order to make it output valid PD patches when a diagram is saved on a file with a .pd extension. This would allow visual impaired users to build a patch – PD programs are dubbed with this name – using the CCmI audio-haptic interface and then save the patch as a .pd file, ready to be open and executed within PD. Since the specifications of how PD patches are saved on files open, it was possible to reproduce the same format in a persistence module of the CCmI editor.
In the time span of a one-day hackathon, only Objects, Numbers and Connections have been implemented and only one connection is can currently be created between inlets and outlets of any two elements. However this achievement opens up the possibility of a full implementation as well as new research questions, such as: how could the hierarchical structure be changed to accommodate a more effective representation of PD patches, based on PD-specific features ?
The code that resulted from the hackathon is available on the SoundSoftware platform, licensed under the GNU general public license.
More on the hierarchical structures that allows for non-visual interaction with diagrams can be found in these publications:
- Metatla, Oussama, Nick Bryan-Kinns, and Tony Stockman. “Interactive hierarchy-based auditory displays for accessing and manipulating relational diagrams.” Journal on Multimodal User Interfaces 5.3-4 (2012): 111-122.
- Metatla, Oussama, Nick Bryan-Kinns, and Tony Stockman. “Constructing relational diagrams in audio: the multiple perspective hierarchical approach.” Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility. ACM, 2008.
Here is some videos of the CCmI Diagram editor in action: