CMD Cross-modal Design
Evan Knowles IDUS-431 | Fall ‘14 | SCAD
Fore word
The concept of Cross Modal Design was conceived while attending classes at SCAD (Savannah College of Art and Design). One afternoon in the studio, a classmate asked “which one looks like salt” while holding up two nearly complete models; the project being salt and pepper grinders inspired by iconic designers. Upon choosing one of the two he exclaimed “that’s what everyone has said” and continued on his way. However, I was left with a question - “How is it the ‘everyone’ had the same answer?”, It was beyond probability, there had to be something else at work. Previously in a color theory class the topic of synesthesia was discussed and I naturally started to draw parallels between the neurological condition and the answer my classmate had received during his impromptu survey.
Table of Contents:
Introduction
1-6
Evidence
7 - 18
Method
19 - 44
Application
45 - 56
Development
57 - 58
Intro duction 1
CMD (Cross-Modal Design), Much like other innovations in design is based in scientific findings and evidence. However, unlike most, CMD is built on evidence from the fields of Neuroscience and Psychology; rather than chemistry, computer science, engineering and the like. A design approach for the consideration of multiple sensory modalities is gaining greater attention in the design ethos. Michael Haverkamp’s “Synesthetic Design For a Multisensory Approach” captures the idea of multisensory design and it’s applications. However based on the principles Haverkamp lays, CMD is in its own right is as much about process and method as it is outcome. CMD is means for designers to use in approaching and applying to overarching schemes that guides design. Designers tend to want a unique but consistent look and feel in their work, and generally try to gain visual attention and object recognition by users. By utilizing CMD in the structing ,or application to, a larger design language designers have the ability to gain multi-modal attention through singular sensory pathways.
2
What is
3
CMD Cross-modal Design
?
CMD is a method to discover ‘Design-Morphemes’ to apply to design(s), creating coherent representations by combining modalities; enabling users to have meaningful perceptual experiences.
4
*
Morpheme:
5
“A meaningful morphological unit of a language that cannot be further divided. A morphological element considered with respect to its functional relations in a linguistic system (e.g., in, come, -ing, forming incoming).”
A ‘Design-Morpheme’ is applied to a greater design language and is used to communicate to users thorough cross-modal perception.
6
Evi dence 7
CMD as a working concept is based on scientific evidence suggesting that designs can impact multiple sensory modalities though individual modality channels, even in users who do not experience overt Synesthesia - [a neurological phenomenon in which stimulation of one sensory or cognitive pathway leads to automatic, involuntary experiences in a second sensory or cognitive pathway].
8
“All of our sensory input (minus olfactory) goes through the thalamus before it makes it’s way to dedicated processing areas in the brain.”
Thalamus
9
The Thalamus A vital structure lying deep within the brain that has several important functions. There are extensive nerve networks that send signals all around the structures of the brain including the cerebral cortex. The thalamus is involved in sensory and motor signal relay and the regulation of consciousness and sleep. Aside from sense of smell, all other sensory processes involve a thalamic nucleus receiving a sensory signal which is then directed to the relevant cortical area. The idea that most sensory information must pass through the thalamus as a sort of cognitive router leads to believe that there is high probability of cross-modal interaction.
Erica R. Knowles, M.S., Neuroscience Doctoral Candidate, Northwestern University CMD Neuroscience Consultant 10
Cross-modal plasticity Adaptive reorganization of neurons to integrate the function of two or more sensory systems. A familiar concept to most, an example may be a blind individual having an enhanced auditory sense; Cross-modal plasticity reworks the network structure of the brain. A paper published by Kupers, Ron, and Maurice Ptito titled “‘Seeing’ Through The Tongue” outlines a process in which cognitively blind participants tongues are electrically stimulated resulting in in the stimulation of their visual cortex
Kupers, Ron, and Maurice Ptito. “‘Seeing’ through the tongue: cross-modal plasticity in the congenitally blind.” International Congress Series. Vol. 1270. Elsevier, 2004. 11
�...the time course of neuroplasticity in humans can be remarkably rapid.�
12
“When different sensory stimuli are only weakly effective, but are spatiotemporally aligned their combined presentation can elicit a response exceeding their sum.�
13
Multisensory Integration Multisensory Intergration is a process by which information from different sensory systems is combined to influence perception, decisions, and overt behavior. This is central to CMD because it allows users to perceive a world of coherent perceptual entities. Also, because multisensory Integration deals with how different sensory modalities interact with one another and alter each other’s processing.
Stein, Barry E., Terrence R. Stanford, and Benjamin A. Rowland. “The neural basis of multisensory integration in the midbrain: its organization and maturation.� Hearing research 258.1 (2009): 4-15. 14
Cross-Modal Perception Cross Modal Perception is the Interactions between two or more different sensory modalities. Cross-modal integration and crossmodal plasticity of the human brain are increasingly studied in neuroscience to gain a better understanding of the large-scale and long-term properties of the brain. Much like multisensory integration CMD is largely dependent on cross-modal perception. Although it has been found that scent information is routed directly the olfactory cortex a study conducted found direct relations to odors and abstract symbols in participants.
Thorsten Sander , Guido Ritter , Thomas Humme, Kerstin Schemmer , Ingeborg Scheer,Han-Seok Seo , Artin Arshamian, Cross-modal integration between odors and abstract symbols, Neuroscience Letters, Accepted 4 May 2010 15
“ The odors generally regarded as being pleasant were paired with circle or curve-shaped symbols. Whereas, the odors judged generally as being unpleasant were paired with square- or angularshaped symbols�
16
17
Synesthetic Design “The Conventional process of design (left-hand side) is primarily based on the separate optimization of product features within the modalities. If undertaken at all, research with respect to cross-sensory correlations is limited to the latter phases of development. Synesthetic design (right-hand side) is primarily focused on possible strategies of connections for which appropriate product features are subsequently selected”
“An optumum design for all senses requires the communication of product features via as many sensory channels as possible” Michael Haverkamp, Synesthetic Design Handbook for a Multisensory Approach, 2013, birkhauser verlag GmbH 18
Meth od 19
CMD uses a four stage process to create validated designmorphemes [Morpheme: “A meaningful morphological unit of a language that cannot be further divided. A morphological element considered with respect to its functional relations in a linguistic system (e.g., in, come, -ing, forming incoming ). A morphological element considered with respect to its functional relations in a linguistic system.” A ‘Design-Morpheme’ is applied to the greater design language, used to communicate and leverage cross-modal perception.] In the test usage of the CMD method Visual X Gustatory morphemes were explored.
20
1.Data Collection
4.Artifact Validation
3.Artifact Creation
21
2.Data Synthesis
Apply Validated Morpheme
22
1 Data Collection no.
In the data collection stage the desired modality channels of activation must be decided upon. Then a means of capturing data must be formulated. In the test usage of CMD a data collection kit Consisting of (1) salt packet (1) pepper packet (1) pen - green (1) collection template (1) small water bottle. The participants were instructed to - open salt packet, take small amount of salt onto tongue. Then given ~ 5 seconds to make a color association and write the first color that comes to mind in the “color association” field on collection template. Once the color association field is completed, a second amount of salt is placed onto tongue and the participant is given ~5 seconds to gestural draw the taste of salt in the “Gesture” field. the process was then repeted using pepper.
23
24
In the test usage of the CMD method 33 undergraduate students and two professors participated in the data collection process.
25
26
Salt Gestures
27
28
Pepper Gestures
Blue, Red, Blue/white, Yellow, Gray, Yellow, Gray, Yellow, White, Lime green, Silver, Yellow-orange, yellow, White, Blue, Blue/white, White, Teal, Yellow, Yellow, Orange, Yellow, Yellow, Yellow, Yellow, White, Red, Yellow, Yellow, White, Blue, Blue, Orange, White, White, Pale yellow
4 Blue
13 Yellow
7 White
2 1 1 1
Light Blue Silver Lime Green Teal
Salt Color Association
29
1 Gray 2 Red
3 Orange
Pepper Color Association
White 2 Black 6
Brown 10
Red 7
Gray 4
Burgundy 3
Red, White, Brown, Red, Brown, Black, Brown, Burgundy, Dark Red, Blue-silver, Orange, Dark orange, Brown, Maroon, Brown, Maroon, Gray, Black, Brown, Brown, Black, Black, Red, Brown, Black, Gray, Black, Brown, Dark red, White, Gray, Red-black, Brown, Gr
Silver-Blue 1 Orange 2
30
no.2
Data Synthesis The data synthesis stage is achieved by pattern identification in the previously collected data. Identifying patterns in the data reveal parameters for artifact creation and later validation. In the test usage of CMD (visual X gustatory) patterns found reveled four gesture pattern groups in each salt and pepper. These patterns identified based on gesture similarities i.e .repeating sharp shapes, repeating soft shapes, shape multiples, and other gesture similarities. Color association were grouped into three categories for each salt and pepper. These patterns were based on the three largest total amounts of color associations in each category by participates in data collection. Color association variants were combined into a singular patterns , i.e. “dark red” and “red” color associations were combined.
31
32
Salt Gesture Patterns
Patterns relating to strictly gesture formations* 33
34
Pepper Gesture Patterns
13 Yellow Total
7 Blue Total
35
Salt Color Patterns
7 White Total
Pepper Color Patterns
10 Brown Total
10 Red Total
6 Black Total
36
no.3
Artifact Creation Artifact creation is determined by data synthesis. Identified patterns are generated into the desired state for application of designmorpheme in final outcome(s). In the test use of CMD (visual X gustatory) the desired state was three dimensional form and with color application. Gesture patterns were assessed and abstract three dimensional form artifacts were created based on each the gesture patterns characteristics. Color pattern artifacts were developed into the state of color swatches.
37
38
Pepper
39
Salt
Salt
Pepper
Salt
Pepper
Pepper
Salt
40
no.4
Artifact Validation Artifact validation is used to assess the validity of patterns identified to create artifacts. This is to ensure an error in synthesis did not render created artifacts useless as design-morphemes. Also to eliminate any artifacts created that do not elicit the desired cross-modal perception in participants. In the test usage of CMD Visual X Gustatory - A group of ten undergraduate students (whom had no previous exposure to the CMD test study) were asked to collaboratively sort the previously generate form artifacts and color swatches dependent on the taste of salt and pepper. An area labeled “Salt” and another labeled “pepper” were placed on a surface. The forms and color swatches were placed in between the two areas in a random assortment. Participants were asked to first taste salt and sort the forms and color swatches. Folled by tasting pepper and reconsider their decision. All of the three color association from each gustatory category were validated as morpheme without any issue in collaboration from participants. One form artifact from each gustatory category (salt - pepper) were sorted into their origin catagory- each of the two forms sorted back into their origin category were generated from the largest gesture pattern in each category. 41
42
*Validated form artifacts; both generated from the largest gesture pattern in each category.
43
Salt Sorting Outcome
*
Pepper Sorting Outcome
*
44
Appli cation 45
Once a validated design-morpheme(s) is discovered it can be applied to the larger design language and functional design. The use of validated CMD design-morpheme(s) is ment to expedite endowment, adoption, and communication of designs to the user(s). Along with creating a coherent representations by combining modalities; enabling users to have meaningful perceptual experiences. These experiences could varry in any cross section of sensory modalities. In the test usage of CMD [Visual X Gustatory] an extremely literal application of design morpheme’s was used the generation in the form and color of a salt and pepper grinder set.
46
Considerations In applying design-morphemes to product from, general design criteria must be taken into consideration. For the test usage of CMD (visual X gustatory) in applying the form morphemes into the product from of a set of salt and pepper grinders considerations of human factors, form factors, and design function were taken into consideration. Human factors - the form must be at a scale for human hands and a reasonable to high level of comfort while using the products must be met. Form factors- the form must accommodate a vessel-area to contain salt crystals and pepper corns, along with housing the milling mechanisms. Design function - the user must be able to grind the contens, refill to vessel-area, and have the ability to adjust coarse/fine grind setting.
47
Human Factors
Form Factors
Design Function
48
Form Development Form development in the test application of CMD (visual X gustatory) was accomplished through applying the validated form morphemes into the product form of salt and pepper grinder set. Finding an appropriate balance of design-morpheme use and design consideration factors was paramount in generating forms that elicited cross-modal perception while also communicating function. Sketch ideation was undertaken by using validated form morphemes to guide the process. Once a final form was uncovered moving into 3d CAD software (solidworks) to build the forms was the following step.
49
50
Color Application Color application in the test usage of CMD (visual X gustatory) was first tested in computerized rendering software (keyshot). Many iterations were undergone to find the correct balanced and appropriate use of validated color design-morphemes. Since all of the color design-morphemes were validated, the color patterns from ‘data synthesis’ were revised to infrom the percentage of color use in each of the products forms. The largest body parts of the product(s) were applied with the largest patter groups and the smallest pattern groups were applied to the smallest product body parts.
51
52
Physical Model Physical modeling in the test application of CMD (visual X gustatory) was done by 3D printing. The CAD models developed were generated in the form development stage. 3d printing was chosen over handbuilding models to remove the chance of human error; which could have possibly obscured the application of design-morphemes in product form(s). 3D printing also allowed for control in maintaining design-morpheme usage in the design intent of the product forms.
53
54
Final Outcome In the final outcome of the test application of CMD (visual X gustatory) form and color design-morphemes were applied to the body design of a of salt and pepper grinder set. They exsit as the physical representation of the taste of what they contain. The physical form and color application combined serve as non-verbal communication. Each of the pieces contents should be, even if subconsciously, understood by users thorough cross-modal perception. The test application, again, was a very literal application of CMD. However, in a first use scenario a literal application severs well to illustrate how CMD as a whole is leveraged and applied.
55
56
Devel opment 57
CMD (Cross Modal Design) has been developed through an individual test useage. The method itself could benifit greatly through further development and definitive, as well as less literial, application. An automated system to collect, sort, and stack data could also have a profound impact on CMD’s efficiency and effectiveness in general; and could revel potential for CMD use in deep-learning applications.
58
Evan Knowles