Intersemiotic Texture: analyzing cohesive devices between language and images

Recent research on multimodal discourse has explored the nature of semantic relations between different semiotic resources. Drawing on the interpretation of language as a social semiotic resource, this article proposes Intersemiotic Texture as the crucial property of coherent multimodal texts and presents a preliminary framework for cohesive devices between language and images. The framework is illustrated through examination of print media to demonstrate how the image–text relations are meta-functionally orchestrated across experiential, textual and logical meanings at the discourse stratum. A discourse-based model is suggested to analyze image–text logical relations complementary to existing grammar-based approaches. This research also develops a meta-language to describe Intersemiotic Cohesive Devices from two complementary perspectives: Intersemiotic Cohesion not only functions to integrate different modes together when multimodal discourse is conceptualized as a finished product, it also constitutes essential text-forming resources for semantic expansions across language and images during the ongoing contextualization of meanings in real time.     

Social distance and the multimodal construction of the Other in sectarian song

This paper introduces the concept of “tonal gravity” through a multimodal analysis of a YouTube video to demonstrate how multimodality is key to the construction of “Rule Britannia” as a sectarian song. The analysis focuses upon the multimodal semiotics of social distance which has been a key concept in sociological and anthropological traditions in recent times. This concept offers a means to understand the social semiotic relationship between Self and Other in multimodal discourse. Following previous work in social semiotics and music studies which examine how visual composition, music and the voice have constructed social distance or expressions of intimacy, I introduce the concept of tonal gravity which extends the metaphors of semiotic space in previous work in the musical mode, to account for a fuller understanding of how music helps narrate the multimodal Self and Other. This is introduced via a close multimodal analysis of “Rule Britannia” as a Rangers Football fan video which is only transformed into a sectarian text through multimodal collocation where different semiotic resources in various modes act in combination to produce a dominant Self actively prejudiced against a low Other.     

Exploring construction project design as multimodal social semiotic practice

ABSTRACT

The semiotic nature of design and engineering communications has often evaded direct interrogation; specifically the representation of requirements using multiple semiotic modes. This paper explores the semiotic character of requirement representation in the construction project design and engineering process by mobilising a social semiotic analytic framework to reveal how project artefacts (e.g. drawings; digital imagery; physical objects) are multimodal resources that critically affect stakeholder engagement and interpretation. The focused empirical study from a hospital construction project reveals the motivations and effects of multimodal and semiotic communicative choices of different parties (e.g. client; architect; engineer; design consultant) engaged on the same project; the representation (or not) of requirements via various semiotic modes being a social and political process. The contribution of the paper is in revealing the semiotic character of requirement representation and the relevance of semiotic-informed enquiry for an industry continuing to develop new technologies (e.g. virtual reality; BIM – building information modelling) for design and construction work.     

Caricatures,cartoons, spoofs and satires: political brands as butts

• A theory of caricature is used to analyse and explain the nature of—and reactions to—a controversial political cartoon depicting Jacob Zuma, the highly publicized president of South Africa, and how the cartoon both played on and affected his political image and brand. Jacob Zuma has crafted his image as a person who has overcome insurmountable injustices to lead his nation. The cartoon directly questioned Zuma's claims of discrimination by the justice system. The cartoonist (Zapiro) was able to use the cartoon to further solidify Zuma's image as an embattled yet contentious and divisive leader. The framework that follows depicts the spoofing of a political brand, in a marketing environment in which parody and spoofing of more conventional products and services are increasingly common.

Copyright © 2009 John Wiley & Sons, Ltd.     

From book to stage to screen: semiotic transformations of Gothic horror genre conventions

ABSTRACT

This paper adopts a multimodal social semiotic approach for exploring the semiotic changes involved in the transformation of a novel into stage and screen productions. It examines how semiotic resources are deployed in each medium through elements of mise-en-scène, such as speech, music, sound, lighting, props, staging, and cinematographic techniques, and the viewing perspectives that are thus established for audiences. The genre of Gothic horror is selected for this purpose, given how this form of performance has transfixed audiences for centuries and has been adapted for both the stage and the screen. In order to demonstrate how each performance medium has produced its own unique set of foregrounding devices to enthral and captivate audiences, a comparative analysis of excerpts from the novel The Woman in Black by Susan Hill, a videotaped theatrical performance, and the 1989 British television film of the same name is undertaken. The paper discusses the implications of the multimodal semiotic approach for developing a better understanding of the semiotic transformations that horror genre conventions undergo in different media and the viewership positions that are thus re-drawn for audiences. The paper concludes with a view of multimodal recontextualisation processes which form the underlying basis of human sociocultural life.     

South Africa after Zuma

South African President Cyril Ramaphosa appears committed to purging the corruption of his predecessor Jacob Zuma, and to rebuilding governmental capacity and improving policy execution. But it will take more than a change in the leadership of the African National Congress (ANC) to restore South Africa's standing as a model of African democracy and continental leadership.     

Social media as semiotic technology and social practice: the case of ResearchGate’s design and its potential to transform social practice

ABSTRACT

This article addresses the need for critical approaches to social media by bridging the focus on language and other semiotic resources that characterises discourse studies with the broader perspective on social media as social, cultural, economic and technological constructs that dominates media and cultural studies. Specifically, we propose a model for analysing how social media as semiotic technologies, that is, technologies designed to enable and constrain meaning-making, may transform social practices. By incorporating Van Leeuwen’s [2008. Discourse and Practice: New Tools for Critical Analysis. London: Oxford University Press] framework for the critical analysis of discourse and social practice, the model extends the social semiotic approach developed in recent critical multimodal studies of software such as PowerPoint to social media, which function primarily to provide platforms for and commodify social practices, rather than to offer rich arrays of semiotic resources for creating multimodal texts and artefacts. Using the academic social network site ResearchGate and the practice of research peer review, we illustrate the model’s capacity to account for the ways the design of social media platforms – through the semiotic resources they make available and the ways these are presented – enables and constrains their users’ ability to perform key social practices and has the potential to transform these practices.     

Studying social media as semiotic technology: a social semiotic multimodal framework

ABSTRACT

How do we study social media technology? While social semiotics provides an extensive toolkit for analysing multimodal texts and semiotic practices, the study of social media as semiotic technology poses a significant challenge to existing research methodologies. In this article, we present a social semiotic framework that allows us to describe in analytical details the multimodal meaning potentials offered by digital social media technology and connect these to multimodal text-making and semiotic practices while underscoring the role of technology. Our framework is organized around seven interrelated and inherently informed dimensions: (1) multimodality, (2) practice, (3) the social, (4) medium, (5) the material, (6) the historical, and (7) the critical. This framework could pertain to most types of semiotic technologies, but will here be developed for accounting for social media technologies, and its viability will be illustrated with examples from Instagram. By developing this framework, we aim at elaborating the theoretical basis and analytical tools of social semiotics, and thereby contributing to bringing forward increased understanding of how social media technology enables making, enacting and managing meaning.     

Chinese tourist site entry tickets: intersemiotic complementarity in an ecosocial process

This article explores how tourist site entry tickets multimodally construe a coherent message and how they are integrated into a grander ecosocial process. Primarily using Royce's intersemiotic complementarity framework we focus on the ideational meanings of an entry ticket to a famous site called the Juyong Pass along the Great Wall of China. Our analysis reveals two types of cohesion: the internal and the extended. As for internal cohesion, intersemiotic complementarity is semantically established through a series of cohesive ties between features of the verbal and the visual modes on the front and the back of the ticket. The extended cohesion is construed through an integration of the multimodal text with the material–physical and other semiotic–discursive practices of its ecosocial environment. During the analysis and discussion, reference is made to another entry ticket to the same site as well as to a database of about 300 entry tickets from tourist sites in China. This article makes an important contribution to an understanding of intersemiotic complementarity in front–back multimodal/multilingual texts and to the relation between this type of complementarity and the concept of ecosocial process.     

A framework for the multimodal analysis of online news galleries: What makes a “good” picture gallery?

The development of digital technology in recent years has led to a revolution in news production and dissemination. In terms of production, we have witnessed a fundamental shift towards visual story-telling. Images dominate the verbal story space and have the potential to become the story themselves. Beyond this, they are also creating unique spaces for themselves (e.g. the online news gallery), with new multimodal genres posing challenges for practitioners and analysts alike. The potential effects of such fundamental shifts on the professional news story-telling practices of the legacy news media provide a rich research opportunity for understanding both how and whether news organisations fulfil their mandate of making sense of the plethora of information that is now available. In this paper, we concern ourselves with one particular innovation in visual news reporting – the online news gallery, or picture gallery in journalism terms. We report on a qualitative analysis of 35 galleries from 12 English-language newspapers with online presence from Europe, North America, Australia and Asia, and interrogate the choices made by institutions in composing this particular type of multimodal “text”, as it is situated in a professional, news story-telling context. In doing so, we explore how a systemic-functional semiotic approach to multimodal news discourse may help us to access the meaning potential of this emerging genre as a vehicle for multimodal digital news reporting, present a framework for the multimodal analysis of online news galleries and consider its implications for the education of media practitioners.     

Analysing multimodality in an interactive digital environment: software as a meta-semiotic tool

The present paper discusses issues arising from the use of an interactive digital software tool to analyse multimodal communication. The focus is on the ways in which such technical resources and associated techniques enable the analyst of social semiosis to apply different types of analysis, and provide the site for critical reflection upon the results of such analyses. The aim is to present an argument for the use of an interactive digital software application as a meta-semiotic tool. Three major challenges for scholars engaged in multimodal social semiotics are addressed in pursuing this aim, specifically with reference to the development of digital interactive analytical resources. Firstly, scholars are faced with the task of not only accounting for an increasing range of semiotic resources, but also for the way different semiotic phenomena interact to produce meaning. Secondly, the multimodal semiotician has a broad range of analytical approaches potentially relevant to any task to draw upon. Thirdly, the nature of contemporary media forms offer challenges themselves in terms of access, analysis and presentation of analysis.     

[De] constructing South Africa's Jacob Zuma led ANC: An Afrocentric perspective

Based on the Afrocentric perspective, the central thesis of this paper is that if the African National Congress (ANC) is to evade the wrath of coalition politics in the near future (year 2019); it needs more than its traditional support base and appeals for liberation rhetoric. Like other liberation movements cum ruling political parties in Southern Africa, indications are that ANC cannot save itself from the brutality of competitive politics and electoral democracy. While the ANC is based in South Africa, it remains the political heritage of all Africans, and as such, it can only survive further degeneration through the collective efforts of pan-Africanists in the country and across the corners of the continent. The fact that President Jacob Zuma led ANC did not have a safe liberalised space for counsel by non-active members has rendered any possible renewal through the support of all Africans a pipe dream. Although the renewal of the ANC may be necessary for the sake of preserving liberation heritage in South Africa and Africa at large, it is concluded that the weakening of dominant political parties such as the ANC is a necessary ingredient for healthy and functioning democracies.     

The clown's carnival in the hospital: a semiotic analysis of the medical clown's performance

The medical clown's multimodal performance in the hospital takes place within its rigid space. A clown in a hospital is a paradox. “Medical clowning” is a metaphor taken from two seemingly unrelated fields of meaning, juxtaposing what is medical, scientific, serious and logical with clowning, emotions, carnival spirit and humour. The clown addresses barriers erected by illness, pain, alienation and distress with a continuous flexible performance of humour and fantasy tailored to changing conditions and circumstances. A clown, by definition, threatens the public order and seemingly has no place in the hospital paradigm. The current article compares medical clowns to carnival clowns, examines the medical clown's flexible performance, illustrates it with case studies, presents a semiotic analysis of the clown's journey through the hospital and examines the significance of the performance on the ideological level.     

Embodiment and meaning: moving beyond linguistic imperialism in social semiotics

Social semiotic approaches to multimodality have tended to take language as the model for other modalities even when their professed aim is to move away from it. This kind of “linguistic imperialism” causes problems for theorising the relationship between the two basic semiotic planes of expression and interpretation in different modalities, and how the affordances of the expression plane relate to the meanings of the interpretation plane in each case, as well as in understanding the particular role of language in multimodal texts. The current paper brings together insights from semiotics, sociology of music and philosophy of language, as well as critiques of social semiotic approaches, in order to argue that the missing element in accounts of semiotic systems like language and music is the fundamental role played by embodiment in both these systems.     

The expanding galaxy of performing arts: extending theories and questioning practices

This paper introduces the Special Issue on the languages of performing arts and is therefore aimed at designing how the context of the latter can be illuminated by socio-semiotic and multimodal approaches to communication. In this Special Issue, performances and performing arts are described as multimodal semiotic acts that co-deploy a range of semiotic resources to produce and construct meanings across different cultures and ages. Seen as dynamic and interactive processes of meaning-making, their analysis calls for new and multidisciplinary frameworks which are collected in this Special Issue. The introduction gives an overview of these papers and discusses their range of diverse phenomena, both live and recorded, including theatre performances and films, art installations, opera, as well as reading out aloud. By outlining the significance and contribution of different disciplines and fields of studies to the broad area of performance studies, the chapter argues the case for innovative approaches that can extend theories and analyse aesthetic and performative practices in context. With the help of some case studies, it provides guidelines for the reading and interpretation of the several theoretical discussions and practical case studies presented to encourage further multidisciplinary research on these domains.     

Semiotics of pride and profit: interrogating commodification in indigenous handicraft production

This study investigates the shifting terrain of pride, profit and power relations in minority language communities under contemporary globalisation. While “pride” associates linguistic-cultural heritage with identity and preservation, “profit” views these as sources of economic gain. In contemporary late capitalism, “pride” seems to be increasingly giving way to “profit”. Arguing that this transformation needs to be interrogated in terms of complexity and that a detailed, multilayered semiotic analysis can open a privileged window for such an inquiry, this study combines critical multimodal discourse analysis and an ethnographic approach to analyse processes of semiotic commodification in handicraft production in the indigenous minority language community of the Sámi in northern Lapland. The investigation focuses on the activities of an innovative Sámi artist and entrepreneur, and within these a range of paper notebooks, which are, although designed by the Sámi artist and sold in her handicraft shop in Lapland, produced by women living at the border of Thailand and Laos. The investigation illuminates two critical shifts: how the move towards profit can open up space to contest the ownership of pride within an ethnic community. Second, how this move makes way for new, globalised modes of production of ‘indigenous handicrafts’ and creates global vectors of power, engaged in both empowerment and exploitation, in the production of both pride and profit. The study thereby contributes to the understanding of the increasingly complex power relations and the ambivalence and multiple effects of practices constituting the apparent shift from “pride” to “profit”.     

Semiotic technologies: a case study of discipline-based practices and pedagogy

This paper examines semiotic technologies, both in terms of the resources they harness and the practices developed around their use. It draws on data collected as part of an ethnographic investigation into the meaning-making practices deployed within civil engineering study. The data is used as a case study for examining semiotic technologies as socially situated resources for disciplinary practices. Using a multimodal social semiotic approach, we argue that technologies are not self-evident, and that their use constitutes specific social practices that require development in the classroom. In order to deploy technologies in pedagogically effective ways, we need to understand the semiotic resources they draw on (including embodied resources). Awareness that technologies are not neutral or value-free, but are socially situated and ideologically laden, may enable meta-level understanding of the discipline, thus creating the possibility for improved pedagogical practices.