Studying social media as semiotic technology: a social semiotic multimodal framework

ABSTRACT

How do we study social media technology? While social semiotics provides an extensive toolkit for analysing multimodal texts and semiotic practices, the study of social media as semiotic technology poses a significant challenge to existing research methodologies. In this article, we present a social semiotic framework that allows us to describe in analytical details the multimodal meaning potentials offered by digital social media technology and connect these to multimodal text-making and semiotic practices while underscoring the role of technology. Our framework is organized around seven interrelated and inherently informed dimensions: (1) multimodality, (2) practice, (3) the social, (4) medium, (5) the material, (6) the historical, and (7) the critical. This framework could pertain to most types of semiotic technologies, but will here be developed for accounting for social media technologies, and its viability will be illustrated with examples from Instagram. By developing this framework, we aim at elaborating the theoretical basis and analytical tools of social semiotics, and thereby contributing to bringing forward increased understanding of how social media technology enables making, enacting and managing meaning.     

Social distance and the multimodal construction of the Other in sectarian song

This paper introduces the concept of “tonal gravity” through a multimodal analysis of a YouTube video to demonstrate how multimodality is key to the construction of “Rule Britannia” as a sectarian song. The analysis focuses upon the multimodal semiotics of social distance which has been a key concept in sociological and anthropological traditions in recent times. This concept offers a means to understand the social semiotic relationship between Self and Other in multimodal discourse. Following previous work in social semiotics and music studies which examine how visual composition, music and the voice have constructed social distance or expressions of intimacy, I introduce the concept of tonal gravity which extends the metaphors of semiotic space in previous work in the musical mode, to account for a fuller understanding of how music helps narrate the multimodal Self and Other. This is introduced via a close multimodal analysis of “Rule Britannia” as a Rangers Football fan video which is only transformed into a sectarian text through multimodal collocation where different semiotic resources in various modes act in combination to produce a dominant Self actively prejudiced against a low Other.     

Social media as semiotic technology and social practice: the case of ResearchGate’s design and its potential to transform social practice

ABSTRACT

This article addresses the need for critical approaches to social media by bridging the focus on language and other semiotic resources that characterises discourse studies with the broader perspective on social media as social, cultural, economic and technological constructs that dominates media and cultural studies. Specifically, we propose a model for analysing how social media as semiotic technologies, that is, technologies designed to enable and constrain meaning-making, may transform social practices. By incorporating Van Leeuwen’s [2008. Discourse and Practice: New Tools for Critical Analysis. London: Oxford University Press] framework for the critical analysis of discourse and social practice, the model extends the social semiotic approach developed in recent critical multimodal studies of software such as PowerPoint to social media, which function primarily to provide platforms for and commodify social practices, rather than to offer rich arrays of semiotic resources for creating multimodal texts and artefacts. Using the academic social network site ResearchGate and the practice of research peer review, we illustrate the model’s capacity to account for the ways the design of social media platforms – through the semiotic resources they make available and the ways these are presented – enables and constrains their users’ ability to perform key social practices and has the potential to transform these practices.     

Exploring construction project design as multimodal social semiotic practice

ABSTRACT

The semiotic nature of design and engineering communications has often evaded direct interrogation; specifically the representation of requirements using multiple semiotic modes. This paper explores the semiotic character of requirement representation in the construction project design and engineering process by mobilising a social semiotic analytic framework to reveal how project artefacts (e.g. drawings; digital imagery; physical objects) are multimodal resources that critically affect stakeholder engagement and interpretation. The focused empirical study from a hospital construction project reveals the motivations and effects of multimodal and semiotic communicative choices of different parties (e.g. client; architect; engineer; design consultant) engaged on the same project; the representation (or not) of requirements via various semiotic modes being a social and political process. The contribution of the paper is in revealing the semiotic character of requirement representation and the relevance of semiotic-informed enquiry for an industry continuing to develop new technologies (e.g. virtual reality; BIM – building information modelling) for design and construction work.     

Semiotic technologies: a case study of discipline-based practices and pedagogy

This paper examines semiotic technologies, both in terms of the resources they harness and the practices developed around their use. It draws on data collected as part of an ethnographic investigation into the meaning-making practices deployed within civil engineering study. The data is used as a case study for examining semiotic technologies as socially situated resources for disciplinary practices. Using a multimodal social semiotic approach, we argue that technologies are not self-evident, and that their use constitutes specific social practices that require development in the classroom. In order to deploy technologies in pedagogically effective ways, we need to understand the semiotic resources they draw on (including embodied resources). Awareness that technologies are not neutral or value-free, but are socially situated and ideologically laden, may enable meta-level understanding of the discipline, thus creating the possibility for improved pedagogical practices.     

The expanding galaxy of performing arts: extending theories and questioning practices

This paper introduces the Special Issue on the languages of performing arts and is therefore aimed at designing how the context of the latter can be illuminated by socio-semiotic and multimodal approaches to communication. In this Special Issue, performances and performing arts are described as multimodal semiotic acts that co-deploy a range of semiotic resources to produce and construct meanings across different cultures and ages. Seen as dynamic and interactive processes of meaning-making, their analysis calls for new and multidisciplinary frameworks which are collected in this Special Issue. The introduction gives an overview of these papers and discusses their range of diverse phenomena, both live and recorded, including theatre performances and films, art installations, opera, as well as reading out aloud. By outlining the significance and contribution of different disciplines and fields of studies to the broad area of performance studies, the chapter argues the case for innovative approaches that can extend theories and analyse aesthetic and performative practices in context. With the help of some case studies, it provides guidelines for the reading and interpretation of the several theoretical discussions and practical case studies presented to encourage further multidisciplinary research on these domains.     

Styling the self online: semiotic technologization in weblog publishing

ABSTRACT

Testing on digital semiotic production the concepts of (self-)styling and technologization of discourse, developed for offline linguistic phenomena, the article investigates the role of digital platforms in shaping the relation between self-expression online, semiotic regulation and the social construction of taste. By focusing on the use of semiotic resources of webdesign for identity expression, the study analyses the semiotic regimes emerging from regulatory practices and webdesign styles on WordPress, and their influence on the changes in the projected identity of a personal blogger. In spite of the participatory character of WordPress, results show the role of the platform in objectifying/technologizing hegemonic semiotic preferences, with consequent normalising effects in bloggers’ self-styling practices. The conclusions relate the findings to broader power dynamics in the social construction of taste and their implications for both online and offline forms of self-expression.     

Intersemiotic Texture: analyzing cohesive devices between language and images

Recent research on multimodal discourse has explored the nature of semantic relations between different semiotic resources. Drawing on the interpretation of language as a social semiotic resource, this article proposes Intersemiotic Texture as the crucial property of coherent multimodal texts and presents a preliminary framework for cohesive devices between language and images. The framework is illustrated through examination of print media to demonstrate how the image–text relations are meta-functionally orchestrated across experiential, textual and logical meanings at the discourse stratum. A discourse-based model is suggested to analyze image–text logical relations complementary to existing grammar-based approaches. This research also develops a meta-language to describe Intersemiotic Cohesive Devices from two complementary perspectives: Intersemiotic Cohesion not only functions to integrate different modes together when multimodal discourse is conceptualized as a finished product, it also constitutes essential text-forming resources for semantic expansions across language and images during the ongoing contextualization of meanings in real time.     

From book to stage to screen: semiotic transformations of Gothic horror genre conventions

ABSTRACT

This paper adopts a multimodal social semiotic approach for exploring the semiotic changes involved in the transformation of a novel into stage and screen productions. It examines how semiotic resources are deployed in each medium through elements of mise-en-scène, such as speech, music, sound, lighting, props, staging, and cinematographic techniques, and the viewing perspectives that are thus established for audiences. The genre of Gothic horror is selected for this purpose, given how this form of performance has transfixed audiences for centuries and has been adapted for both the stage and the screen. In order to demonstrate how each performance medium has produced its own unique set of foregrounding devices to enthral and captivate audiences, a comparative analysis of excerpts from the novel The Woman in Black by Susan Hill, a videotaped theatrical performance, and the 1989 British television film of the same name is undertaken. The paper discusses the implications of the multimodal semiotic approach for developing a better understanding of the semiotic transformations that horror genre conventions undergo in different media and the viewership positions that are thus re-drawn for audiences. The paper concludes with a view of multimodal recontextualisation processes which form the underlying basis of human sociocultural life.     

Embodiment and meaning: moving beyond linguistic imperialism in social semiotics

Social semiotic approaches to multimodality have tended to take language as the model for other modalities even when their professed aim is to move away from it. This kind of “linguistic imperialism” causes problems for theorising the relationship between the two basic semiotic planes of expression and interpretation in different modalities, and how the affordances of the expression plane relate to the meanings of the interpretation plane in each case, as well as in understanding the particular role of language in multimodal texts. The current paper brings together insights from semiotics, sociology of music and philosophy of language, as well as critiques of social semiotic approaches, in order to argue that the missing element in accounts of semiotic systems like language and music is the fundamental role played by embodiment in both these systems.     

Defensive semiotic strategies in government: a multimodal study of blame avoidance

In this article, I argue that when officeholders try to avoid blame, they often engage in defensive performances which can be analysed in terms of how they exploit various multimodal semiotic resources to ward off (potential) criticism: the setting, appearance, and manner chosen for a particular interaction with an audience; and both verbal and non-verbal ways of arguing, legitimising, framing, and representing social actors and actions. I apply these analytic categories to interpret the data gathered during fieldwork at a major training event of British government communicators. By combining insights from multimodal discourse analysis, discourse-historical studies of organisational behaviour, and recent research into blame avoidance in public administration, I demonstrate how certain semiotic strategies used by officeholders have an effect of backgrounding the ideas about any possible norm violations that government communicators may have been associated with in the eyes of critical audiences, such as lying, spin doctoring, and using tax money for propaganda campaigns that may not actually serve the interests of the public. I suggest that analysts of government communication should pay more attention to the defensive practices of government insiders, and study in great detail how these practices are incorporated into everyday behaviour through professional training.     

The resemioticisation of the socio-cultural construct of Nowruz festival in Tehran’s “Fatemi Spring” billboards

This study explores how the socio-cultural aspects of Nowruz are resemiotised to represent, orient and organize ideological constructs in Tehran’s “Fatemi Spring” billboards. These billboards are semiotic constructs with mixed messages about the coincidence of the Iranian new year festival of Nowruz at the beginning of spring and the Shia religious mourning on the occasion of the passing away of the holy prophet’s daughter Fateme, known as the Fatemiye period. Based on an integrated social semiotics approach, the study draws on multimodal analysis to elucidate the way semiotic resources are used for ideological transition within this discursive context. Quantitative enquiry methods are applied to illuminate the meaning construction processes shaped by the research corpus. The results show that in the generic discourse of Fatemi Spring billboards, the elements of Fatemiye mourning that belong to Shia beliefs are mostly preferred and the natural elements of spring are replaced by the Nowruz celebration rituals. Thus, people in power (billboard designers) appear to apply these ideological structures as a means to exercise discursive control over Tehran citizens.     

A framework for the multimodal analysis of online news galleries: What makes a “good” picture gallery?

The development of digital technology in recent years has led to a revolution in news production and dissemination. In terms of production, we have witnessed a fundamental shift towards visual story-telling. Images dominate the verbal story space and have the potential to become the story themselves. Beyond this, they are also creating unique spaces for themselves (e.g. the online news gallery), with new multimodal genres posing challenges for practitioners and analysts alike. The potential effects of such fundamental shifts on the professional news story-telling practices of the legacy news media provide a rich research opportunity for understanding both how and whether news organisations fulfil their mandate of making sense of the plethora of information that is now available. In this paper, we concern ourselves with one particular innovation in visual news reporting – the online news gallery, or picture gallery in journalism terms. We report on a qualitative analysis of 35 galleries from 12 English-language newspapers with online presence from Europe, North America, Australia and Asia, and interrogate the choices made by institutions in composing this particular type of multimodal “text”, as it is situated in a professional, news story-telling context. In doing so, we explore how a systemic-functional semiotic approach to multimodal news discourse may help us to access the meaning potential of this emerging genre as a vehicle for multimodal digital news reporting, present a framework for the multimodal analysis of online news galleries and consider its implications for the education of media practitioners.     

Business virtual system in the context of e‐governance: Investigation of secure access to information resources

Contemporary digital age proposes different opportunities in the public, social, private, and business areas based on remote access to distributed resources placed in virtual, cloud, educational, governmental, business, smart, and so forth environments. In this reason, the e‐governance is realized on the basis of digital communications between different parties that requires strong protection of the information resources and personal data. The digital sustainability is an important task for e‐governance in the digital world. Article deals with principles of correct protection of the information resources (including in the cloud) by using strong procedures for access regulation of the external clients. Organization of Information Security Management System for identification, registration, authentication, and authorization of the requests is discussed. An investigation of the proposed functionality is made on the basis of stochastic analytical modelling by using the apparatus of Markov chains and statistical processing and interpretation of the obtained results.     

Multimodal narratives in SIA's “Singapore Girl” TV advertisements – from branding with femininity to branding with provenance and authenticity?

This paper analyses the multimodal narratives in Singapore Airlines (SIA)’s TV advertisements. Adapting the concept of the foreground–background continuum in multimodal interaction analysis, the first section of this paper analyses the narratives in 10 TV commercials available online from 1970s to 2013. Following the comparative analysis, a close investigation from a social semiotic perspective is conducted on SIA's latest TV advertisement in 2013. Through the analysis, this study shows how gendered representations of the Singapore Girl are manipulated in SIA's advertisements. It also reveals how these gendered representations adopted by SIA are married with contemporary branding philosophies, which also work to cloak them in relation to the use of increasingly well-rounded narrative structure. This study hopes to shed light on the understanding of how gender and sexist ideologies and representations are discursively embodied in contemporary branding strategies used by large-scale advertisements and campaigns spanning a long period.     

Extending resemiotisation: time,space and body in discursive representation

The present article offers a critical appraisal of resemiotisation through the investigation of a controversial comment made by South African president, Jacob Zuma. The comment (that Jacob Zuma took a shower to minimise his risk of contracting HIV/AIDS) was made during a rape trial in 2006. Over a period of eight years, this comment has been continually recontextualised into different modes, and has served different functions. This article investigates newspaper reports, cartoons and YouTube clips which were collected from 2006 to 2014. All of these texts refer to the “Shower comment” in linguistic, visual or multimodal ways. Drawing on a multimodal analysis, focussing specifically on modes of representation, the article critically discusses the notions of mode, time and space in connection to resemiotisation. I argue that a Bakhtinian account of time, space and body allows for a non-binary view of semiotic phenomena that will lead to a deeper theoretical understanding of semiotic transformations in all their complexity.     

mStories: exploring semiotics and praxis of user-generated mobile stories

Innovations in information and communication technologies have allowed people to actively author multimodal content and engage in new meaning-making practices. New Literacies research has gone some way to understanding new meaning-making behaviours. However, this research often derives its understandings from studies undertaken with students enrolled in formal educational settings. Mobile technologies are increasingly situated outside such domains; the informal use of these devices by adults remains on the periphery of scholarly focus. mStories is a creative participatory digital mobile storytelling project. Taking a multidimensional perspective, this article presents the in-depth case analysis of one participant and their mStory. A semiotic analysis found that the user-generated content demonstrated complex and sophisticated multimodal sense relations. However, control over the textual or compositional meta-function of the text was determined largely by the computer interface, with users habituated to relinquishing authorial control over this element. Within this study, mobile literacy praxis was characteristically ad hoc and contextually embedded, and though mobile technology invites such practices, users were neither determined nor limited by this, and happily turned to other devices where necessary.     

Analysing characters' interactions in filmic text: a functional semiotic approach

This paper presents an analytical tool for constructing characters' action patterns in film. Understanding characters in moving images has been regarded as one of the most significant elements of narrative comprehension and interpretation because spectators' inferences of narrative structures are substantially mediated by characters. The social semiotic method presented in this paper shows how characters' actions and interactions can be systematically constructed based on the co-patterning of textual elements in film and how the patterns provide an analytical foundation for understanding and interpreting characters. In addition, this paper also reviews the previous explorations of the more fine-grained approaches to character analysis and then demonstrates how the method the paper suggests can support and strengthen the accounts provided to date. Most importantly, by comparing scenes within and across two war films, The Thin Red Line (1998) and Black Hawk Down (2001), the article will show the potential of this method for effectively constraining hypotheses for empirical investigations of filmic meaning.