Intersemiotic Texture: analyzing cohesive devices between language and images

Recent research on multimodal discourse has explored the nature of semantic relations between different semiotic resources. Drawing on the interpretation of language as a social semiotic resource, this article proposes Intersemiotic Texture as the crucial property of coherent multimodal texts and presents a preliminary framework for cohesive devices between language and images. The framework is illustrated through examination of print media to demonstrate how the image–text relations are meta-functionally orchestrated across experiential, textual and logical meanings at the discourse stratum. A discourse-based model is suggested to analyze image–text logical relations complementary to existing grammar-based approaches. This research also develops a meta-language to describe Intersemiotic Cohesive Devices from two complementary perspectives: Intersemiotic Cohesion not only functions to integrate different modes together when multimodal discourse is conceptualized as a finished product, it also constitutes essential text-forming resources for semantic expansions across language and images during the ongoing contextualization of meanings in real time.     

Doing (things with) sounds: introduction to the special issue

This special issue aims at analyzing music as a site of social semiosis, i.e., at investigating the manifold ways in which music is constituted as a socially shared event. The papers collected here follow three main threads, considered as central aspects of music making: studying the kind of coordination and participation required to make music together; looking at the semiotic resources employed by musicians to construct their roles in interaction; examining the relationship between language and music. A variety of perspectives is adopted, ranging from social semiotics to conversation analysis, anthropology, multimodal analysis, and critical discourse analysis. Such a variety is also reflected in the musical traditions – Western art music, jazz, gospel, church hymns, pop music – and in the settings under examination, which comprise instructional activities like musical classes and rehearsals, as well as ordinary conversations and written accounts of musicians' biographies. Issues of epistemicity and authority, intersubjectivity, correction of musical action, solidarity, and ideology are thereby addressed. The issue thus aims at exploring the richness and complexity of music making as a social practice, and documents how the integration of different disciplinary perspectives can offer fruitful insights on music as a domain of sociality which lies at the intersection of aurality and writing, norm and creativity, individuality and collectiveness.     

The expanding galaxy of performing arts: extending theories and questioning practices

This paper introduces the Special Issue on the languages of performing arts and is therefore aimed at designing how the context of the latter can be illuminated by socio-semiotic and multimodal approaches to communication. In this Special Issue, performances and performing arts are described as multimodal semiotic acts that co-deploy a range of semiotic resources to produce and construct meanings across different cultures and ages. Seen as dynamic and interactive processes of meaning-making, their analysis calls for new and multidisciplinary frameworks which are collected in this Special Issue. The introduction gives an overview of these papers and discusses their range of diverse phenomena, both live and recorded, including theatre performances and films, art installations, opera, as well as reading out aloud. By outlining the significance and contribution of different disciplines and fields of studies to the broad area of performance studies, the chapter argues the case for innovative approaches that can extend theories and analyse aesthetic and performative practices in context. With the help of some case studies, it provides guidelines for the reading and interpretation of the several theoretical discussions and practical case studies presented to encourage further multidisciplinary research on these domains.     

Social distance and the multimodal construction of the Other in sectarian song

This paper introduces the concept of “tonal gravity” through a multimodal analysis of a YouTube video to demonstrate how multimodality is key to the construction of “Rule Britannia” as a sectarian song. The analysis focuses upon the multimodal semiotics of social distance which has been a key concept in sociological and anthropological traditions in recent times. This concept offers a means to understand the social semiotic relationship between Self and Other in multimodal discourse. Following previous work in social semiotics and music studies which examine how visual composition, music and the voice have constructed social distance or expressions of intimacy, I introduce the concept of tonal gravity which extends the metaphors of semiotic space in previous work in the musical mode, to account for a fuller understanding of how music helps narrate the multimodal Self and Other. This is introduced via a close multimodal analysis of “Rule Britannia” as a Rangers Football fan video which is only transformed into a sectarian text through multimodal collocation where different semiotic resources in various modes act in combination to produce a dominant Self actively prejudiced against a low Other.     

Correction sequences and semiotic resources in ensemble music workshops: the case of Conduction®

The paper examines the phenomenon of correction of musical action by analysing ensemble music workshops devoted to Conduction®, a way of making music together on the basis of a codified gestural lexicon addressed by a conductor to instrumentalists. In particular, it focuses on correction sequences initiated by the conductor and related to the musical material to be played and discusses the techniques used by the conductor to construct and sequentially organize correction, showing how these are accomplished through the interplay and mutual contextualization of talk and further audible and visible semiotic resources. Based on the fine-grained analysis of two correction sequences, the study stresses the role of multimodality for investigating music and pedagogy from an interactional perspective, while highlighting the peculiarities of Conduction as performative and educational practice.     

Social media as semiotic technology and social practice: the case of ResearchGate’s design and its potential to transform social practice

ABSTRACT

This article addresses the need for critical approaches to social media by bridging the focus on language and other semiotic resources that characterises discourse studies with the broader perspective on social media as social, cultural, economic and technological constructs that dominates media and cultural studies. Specifically, we propose a model for analysing how social media as semiotic technologies, that is, technologies designed to enable and constrain meaning-making, may transform social practices. By incorporating Van Leeuwen’s [2008. Discourse and Practice: New Tools for Critical Analysis. London: Oxford University Press] framework for the critical analysis of discourse and social practice, the model extends the social semiotic approach developed in recent critical multimodal studies of software such as PowerPoint to social media, which function primarily to provide platforms for and commodify social practices, rather than to offer rich arrays of semiotic resources for creating multimodal texts and artefacts. Using the academic social network site ResearchGate and the practice of research peer review, we illustrate the model’s capacity to account for the ways the design of social media platforms – through the semiotic resources they make available and the ways these are presented – enables and constrains their users’ ability to perform key social practices and has the potential to transform these practices.     

“They might read a fly speck”: musical literacy as a discursive resource in Louis Armstrong's autobiographies

Jazz has been described as a music in which the “oral” element plays a crucial role, in opposition to Western “classical” music, seen as a chiefly “written” tradition. Although such an image is frequently advocated by critics and musicians themselves, it is also true that it can generate ambivalence and negative outputs, such as the persistent myth of “primitivism” and “naivety,” often associated with jazz music. Building on Social Semiotics and Critical Discourse Analysis, this study aims at analyzing how the representations of “orality” and “literacy,” that emerge in some autobiographical narratives by Louis Armstrong, are generated, and how they can work as semiotic and discursive resources. It argues that the different depictions of musicians, and the attitude displayed toward musical literacy, are sensitive to the historical, societal, and political context in which texts have been produced and published, as well as to the narrator's willingness and ability to resist or subvert dominant discourses. Moreover, the characterization of a musician (or a category of musicians) as able or unable to access musical literacy can also serve local purposes, such as expressing the narrator's stance toward narrative characters.     

Interdependence,interaction and metaphor in multisemiotic texts

Systemic frameworks for discourse analysis of visual/verbal and mathematics texts are used to examine the functionality of language? visual display and mathematical symbolism, and the meaning arising from interaction and interdependence between these semiotic codes in joint constructions. The nature of the interaction between options, classified as unmarked or marked, determines the meaning made in that instance and, more generally, provides an impetus for the expansion of the meaning potential of each semiotic. The phenomenon where semantic shifts occur when functional elements are reconstrued in another semiotic is called semiotic metaphor. Rather than the notion of an overlay of meaning typically associated with grammatical metaphor, the potential exists here for more dramatic forms of semantic shift. Examples of semiotic metaphor categorised as ‘parallel semiotic metaphor’ and ‘divergent semiotic metaphor’ are given in the text analysis.     

Mapping Cultural Auracy: The Sonic Politics of The Day the Earth Stood Still

Anna Kassabian writes in Hearing Film that “classical Hollywood film music is a semiotic code, and that it can and should be subjected to various semiotic and cultural studies methods, such as discourse analysis and ideology critique” (p. 36). This paper examines the sound of a particular Hollywood film—the B-Grade 1950 science fiction “classic” The Day the Earth Stood Still (1951)—in order to perform the kind of analysis Kassabian demands; but also to argue that the analysis needs to encompass not only music, but all sonic elements of the film. Furthermore, the paper argues for development of a cultural auracy that will complement studies of verbal and visual literacies in multimodal and multimedia texts.     

Exploring the complementary effect of post-structuralism on sociocultural theory of mind and activity

This article explores the methodological value of post-structuralism for sociocultural theory. The Peircean notion of knowledge serves as a rationale for incorporating a post-structuralist perspective of discourse into Vygotsky-inspired sociocultural approach to conceptual development. Central to this incorporation is an analytic frame of reference devised to examine how knowledge construction is mediated through the use of language and other semiotic tools. The article provides an interpretive analysis of mother–child interaction in the reading of dual-language picture books, interwoven with a post-structuralist exposition of knowledge, power and discourse. It is suggestive of what sociocultural theory would need to consider in terms of power relations when theorising the role of semiotic mediation in transforming human mind and activity.     

Styling the self online: semiotic technologization in weblog publishing

ABSTRACT

Testing on digital semiotic production the concepts of (self-)styling and technologization of discourse, developed for offline linguistic phenomena, the article investigates the role of digital platforms in shaping the relation between self-expression online, semiotic regulation and the social construction of taste. By focusing on the use of semiotic resources of webdesign for identity expression, the study analyses the semiotic regimes emerging from regulatory practices and webdesign styles on WordPress, and their influence on the changes in the projected identity of a personal blogger. In spite of the participatory character of WordPress, results show the role of the platform in objectifying/technologizing hegemonic semiotic preferences, with consequent normalising effects in bloggers’ self-styling practices. The conclusions relate the findings to broader power dynamics in the social construction of taste and their implications for both online and offline forms of self-expression.     

From book to stage to screen: semiotic transformations of Gothic horror genre conventions

ABSTRACT

This paper adopts a multimodal social semiotic approach for exploring the semiotic changes involved in the transformation of a novel into stage and screen productions. It examines how semiotic resources are deployed in each medium through elements of mise-en-scène, such as speech, music, sound, lighting, props, staging, and cinematographic techniques, and the viewing perspectives that are thus established for audiences. The genre of Gothic horror is selected for this purpose, given how this form of performance has transfixed audiences for centuries and has been adapted for both the stage and the screen. In order to demonstrate how each performance medium has produced its own unique set of foregrounding devices to enthral and captivate audiences, a comparative analysis of excerpts from the novel The Woman in Black by Susan Hill, a videotaped theatrical performance, and the 1989 British television film of the same name is undertaken. The paper discusses the implications of the multimodal semiotic approach for developing a better understanding of the semiotic transformations that horror genre conventions undergo in different media and the viewership positions that are thus re-drawn for audiences. The paper concludes with a view of multimodal recontextualisation processes which form the underlying basis of human sociocultural life.     

Exploring construction project design as multimodal social semiotic practice

ABSTRACT

The semiotic nature of design and engineering communications has often evaded direct interrogation; specifically the representation of requirements using multiple semiotic modes. This paper explores the semiotic character of requirement representation in the construction project design and engineering process by mobilising a social semiotic analytic framework to reveal how project artefacts (e.g. drawings; digital imagery; physical objects) are multimodal resources that critically affect stakeholder engagement and interpretation. The focused empirical study from a hospital construction project reveals the motivations and effects of multimodal and semiotic communicative choices of different parties (e.g. client; architect; engineer; design consultant) engaged on the same project; the representation (or not) of requirements via various semiotic modes being a social and political process. The contribution of the paper is in revealing the semiotic character of requirement representation and the relevance of semiotic-informed enquiry for an industry continuing to develop new technologies (e.g. virtual reality; BIM – building information modelling) for design and construction work.     

A rhetoric of style: Eleanor Rigby and the reordering of popular music

In this paper, we present a synthesis of Kenneth Burke’s rhetoric of identification and Jay Lemke’s social semiotics to frame Eleanor Rigby by the Beatles as a unique point in the phylogenesis of recorded popular music. We emphasise the social semiotic functioning of string arrangements as styles, with style also being understood in the manner of Burke, and style names and definitions being drawn from a corpus analysis of string arrangements for popular music. We argue that, through a rhetoric of style, Eleanor Rigby made canonical claims against rock's cultural counterpart, classical music. We demonstrate the working of the rhetoric and its political implications in the context of the counter-cultural forces active during the mid-1960s.     

How society creates musical styles: jazz and ritual insults

Many of the ways in which music can be used to create meaning for its consumers have been the subject of existing social semiotic literature. However, less is known about how social and cultural environments impact the development of musical sounds. This study uses Tagg’s sign typology as a framework for identifying meaningful signs such as form, responsoriality, and melodic complexity in three jazz bebop recordings by Charlie Parker. It is argued that the values of individuality, communication, and competition expressed by bebop music may be reflections of a sociolinguistic phenomenon known as ritual insults, thus demonstrating one way in which culture affects the creation of musical styles.     

A Theoretical and Analytical Synthesis of Autopoiesis and Sociolinguistics for the Study of Organisational Communication

The purpose of this paper is threefold. First, we propose a systemic view of communication based in autopoiesis, the theory of living systems formulated by Maturana & Varela (1980, 1987). Second, we show the links between the underpinning assumptions of autopoiesis and the sociolinguistic approaches of Halliday (1978), Fairclough (1989, 1992, 1995) and Lemke (1995, 1994). Third, we propose a theoretical and analytical synthesis of autopoiesis and sociolinguistics for the study of organisational communication. In proposing a systemic theory for organisational communication, we argue that traditional approaches to communication, information, and the role of language in human organisations have, to date, been placed in teleological constraints because of an inverted focus on organisational purpose-the generally perceived role of an organisation within society-that obscure, rather than clarify, the role of language within human organisations. We argue that human social systems are, according to the criteria defined by Maturana and Varela, third-order, non-organismic living systems constituted in language. We further propose that sociolinguistics provides an appropriate analytical tool which is both compatible and penetrating in synthesis with the systemic framework provided by an autopoietic understanding of social organisation.     

Mutant Enunciations: Feminist Practices and the Anoedipal

Feminist theory and practice, and in particular Julia Kristeva's theory of the semiotic in language and Bracha Lichtenberg Ettinger's explication of the matrixial, provide a basis from which to extend the ideas of Deleuze and Guattari concerning domains of signification that operate beyond the Oedipal structure. In the first part of this paper, I will discuss some relationships that may be traced between the ideas of these theorists and processes in creative practice that have the potential to constitute an alternative stratum of subjectivity not wholly dependent on Oedipalisation. This will be followed by a brief analysis of two paintings by West Australian artist Linda Banazis incorporating what I have termed the ‘X Function', an additional and transversal system to those outlined in Michael O'Toole's (1994) model of semiotic analysis of visual texts. The X Function provides a means of mapping nomadic or non-Oedipal currents, those ‘non-discursive' or resistant dimensions of texts which deny fixity or closure of interpretation.