Facial Muscles

This paper presents a re-analysis of data we reported previously (Hipp et al., 2011 (link)). We recorded the continuous EEG from 126 scalp sites and the electrooculogram (EOG) from two sites below the eyes all referenced against the nose tip (sampling rate: 1000 Hz; high-pass: 0.01 Hz; low-pass: 250 Hz; Amplifier: BrainAmp, BrainProducts, Munich, Germany; Electrode cap: Electrodes: sintered Ag/AgCl ring electrodes mounted on an elastic cap, Falk Minow Services, Herrsching, Germany). Electrode impedances were kept below 20 kΩ. Offline, the data were high-pass filtered (4 Hz, Butterworth filter of order 4) and cut into trials of 2.5 s duration centered on the presentation of the sound (−1.25 to 1.25 s). First, trials with eye movements, eye blinks, or strong muscle activity were identified by visual inspection and rejected from further analysis (trials retained for further analyses n = 345 ± 50, mean ± s.d.). Next, we used independent component analysis (FastICA, http://www.cis.hut.fi/projects/ica/fastica/; Hyvärinen, 1999 (link)) to remove artifactual signal components (Jung et al., 2000 (link); Keren et al., 2010 (link)). The removed artifactual components constituted facial muscle components (n = 45.8 ± 7.84, mean ± s.d.), microsaccadic artifact components (n = 1.2 ± 0.82, mean ± s.d.), auricular artifact components (O'Beirne and Patuzzi, 1999 (link)) (n = 0.5 ± 0.83, mean ± s.d.), and heart beat components (n = 0.5 ± 0.59, mean ± s.d.). Alternatively to ICA, we accounted for microsaccadic artifacts by removing confounded data sections identified in the radial EOG using the approach and template described in Keren et al. (2010 (link)) (Threshold: 3.5). Importantly, for this analysis step, we did not reject entire trials containing a microsaccadic artifact (79 ± 18%, mean ± s.d., of trials contained at least one saccadic spike artifact), but only invalidated the data in the direct vicinity of detected artifacts (±0.15 s). Whenever the window for time-frequency transform overlapped with invalidated data (see spectral analysis below), it was rejected from further analysis. As a consequence, spectral estimates were based on varying amount of data across time and frequency. We derived the radial EOG as the difference between the average of the two EOG channels and a parietal EEG electrode at the Pz position of the 10–20-system. Notably, rejection based on the radial EOG may miss saccadic spike artifacts of small amplitude that can be detected with high-speed eyetracking (Keren et al., 2010 (link)). However, the fact that we did not find any significant saccadic spike artifacts after radial EOG based rejection at those source locations that before cleaning best captured these artifacts (cf. Figure 7C) suggests that potentially remaining artifacts are small.

Concerning the specificity of elaboration, the procedure was conducted in a few stages. In the first stage, so-called “face banks” were used, and were gathered by an agency specializing in the recruitment of actors, extras, etc.. Native Polish applicants (n = 120) aged 20–30 were chosen and invited to individual meetings: 30-min photography sessions during which the expected work results were discussed. All of the participants provided signed consent to the recruitment agency and agreed to participate in the project before arriving to the laboratory. After the meeting they were informed that if they did not want to participate they are not obligated to continue the cooperation, and none of the taken photographs will be used in the future. The meetings also aimed to choose participants who had acting experience and positive motivation toward cooperation. After this stage, 60 people were selected and invited to take part in the project.
Natural emotional expression is involuntary, and bringing it under any control requires exercises allowing activation of particular muscles. Therefore, a set of exercises based on an actor training system, the Stanislavski method (1936/1988 ), was developed. The aim was to maximize the authenticity of expressed emotions in photography session conditions. The method assumes that realistic presentation of a given emotion should be based on the concept of emotional memory as a point of focus. Thus, actors must first recall an event when he or she felt given emotion, and then recall physical and psychological sensation. Additionally, this technique includes a method of “physical actions” in which emotions are produced through the use of actions. To do so, the actor performs a physical motion or a series of physical activities to create the desired emotional response for the character. For instance, feeling and expressing sadness presumes recalling sad personal experiences as well as some physical action, e.g., sighing and holding head in hands. The training consisted of three parts: (1) Theoretical presentation about the physiology of emotion and mimic expressions and demonstration of key elements essential for gaining a desired expression. During this stage, participants were also presented the theoretical foundations concerning the creation of the set, which facilitated understanding of the authors' intentions and communication during photography sessions. (2) Training workshops taking place under supervision of the set's authors, and (3) Training and exercises “at home.” The final stage was the photography session during which photographs of practiced expressions were registered. During the sessions, participants were first allowed to focus on the given emotion (so perform exercises evoking emotion) and then show facial expression of felt emotion to the camera. No beards, mustaches, earrings or eyeglasses, and preferably no visible make-up was allowed during this stage, yet matting foundation make-up was applied to avoid skin reflections.
After gathering the photographic material, primary evaluation was conducted. Photographs with appropriate activity of muscular/facial units were selected. Facial unit activity was specified according to descriptions of the FACS (Ekman et al., 2002 ) and characteristics of visible facial changes caused by contraction or relaxation of one or more muscles (so called Action Units) during emotional expression (see Table 1 for details). At this stage, approximately 1000 photographs of 46 people were selected by our competent judges. The photographs were subjected to digital processing for format, resolution, and tone standardization. Pictures were initially cropped and resized to 1725 × 1168 pixels and the color-tone was balanced for a white background.

Footage from 28 privately owned dogs of varying breeds (approximately 8–10 hours) from the Max Planck Institute for Evolutionary Anthropology DogLab was the primary source for DogFACS development. In addition, we sourced approximately 100 clips from www.youtube.com (permission granted from the copyright holder of each clip) and used ad hoc footage from 86 dogs at four dog shelters (Portsmouth City Dog Kennels; Wood Green, The Animal’s Charity in Cambridge; The Dog’s Trust, West London, Harefield and RSPCA Southridge Animal Centre, London). Each facial movement was documented by appearance changes, minimal criteria for identification and comparison to other species, in line with FACS terminology (Table 1). The muscular basis of each facial movement was verified in light of dissection of a face from a specimen of a domestic dog (AMB) as well as previously published dissections [25] . The manual is freely available and requires certification to use (www.dogfacs.com).

Raw optical density variations were acquired at three wavelengths of light (780, 805 and 830 nm), which were translated into relative chromophore concentrations using a Beer–Lambert equation [84 (link)–86 (link)]. Signals were recorded at 30 Hz. Baseline drift was removed using wavelet detrending provided in NIRS-SPM [51 (link)]. In accordance with recommendations for best practices using fNIRS data [87 (link)], global components attributable to blood pressure and other systemic effects [88 (link)] were removed using a PCA spatial global mean filter [60 (link),62 (link),89 (link)] before general linear model (GLM) analysis. This study involves emotional expressions that originate from specific muscle movements of the face, which may cause artefactual noise in the OxyHb signal. To minimize this potential confound, we used the HbDiff signal, which combines the OxyHb and deOxyHb signals for all statistical analyses. However, following best practices [87 (link)], baseline activity measures of both OxyHb and deOxyHb signals are processed as a confirmatory measure. The HbDiff signal averages are taken as the input to the second level (group) analysis [90 (link)]. Comparisons between conditions were based on GLM procedures using NIRS-SPM [51 (link)]. Event epochs within the time series were convolved with the haemodynamic response function provided from SPM8 [91 ] and fitted to the signals, providing individual ‘beta values’ for each participant across conditions. Group results based on these beta values are rendered on a standard MNI brain template (TD-ICBM152 T1 MRI template [72 (link)]) in SPM8 using NIRS-SPM software with WFU PickAtlas [73 (link),74 (link)].

This study included 11 motions. Figure 1 shows the relation between the different motions and the techniques we used. Different methods were used for different motions based on the characteristics of these motions. OpenFace (Baltrušaitis et al., 2016 ) was used for extracting features of FE. The facial-action-coding system defines the correspondence between facial emotions and facial muscles and divides facial expressions into 46 action units (AU). OpenPose (Cao et al., 2021 (link)) was used for pose estimation, which can provide estimates of the position of 25 2D points of the human body and 21 2D points of the hand. OpenPose was used for motions including FT, HM, TT, LA, AFC, GAIT, and POS. Since some estimation of motions by OpenPose related to joints could cause inaccurate result when asking patients to straighten their arms (including PSOH, PTOH, and KTOH). For alleviating this impact, instead of OpenPose, we used HRNet (Wang et al., 2020 (link)) to increasing the accuracy during joint estimation.
In regard to the relationships between motions and dependent variables, features for the Rig-UE model were from seven motions related to the upper extremities, including FT, HM, PSOH, AFC, PTOH, KTOH, and GAIT. Features for the Rig-LE model were from four motions related to the lower extremities, including LA, AFC, and GAIT. In addition to the features used in the Rig-LE model, the PS model included one additional POS motion, i.e., the standing posture, which can represent the balance of the participant while remaining still. For the Rig-Neck model, we used all the motions.
Parkinson’s disease impacts the left and right limb of patients. Especially for PD of early stage, these patients can have significantly severe limb side, which means directly using feature of left or right can cause problem of inconsistency. To remove the impact of the severe limb side, we calculated some parameters using the two sides of the extremities to represent the overall condition or calculated the difference between the two sides of the participant. For a specific feature related to the two sides, for example, the release speed during FT, we calculated the mean, maximum, and minimum values for both hands and the absolute difference between right hand and left hand.

We used multilevel models to assess the effect of facial expression activation on pain categorization and pain intensity estimates and whether target race, gender, group membership, or similarity had an impact on these effects. We evaluated maximal models (Barr et al., 2013 ) using both the lmer and glmer functions in the lme4 package in R (Bates et al., 2015 (link)). Although our similarity measure allowed for more nuance compared to group membership (i.e., a perceiver can have varying levels of similarity towards targets that may be equivalent by group membership), our factors for similarity and group membership had high multicollinearity; therefore, we evaluated each factor in separate models.
We used logistic multilevel models in each study to assess the odds a trial was rated as painful or not-painful based on expression intensity (i.e., do greater facial expression activations increase the odds of rating a trial as painful) and whether any of our sociocultural measures impacted this relation. If a participant did not have at least three trials in which they observed pain and three trials in which they observed no-pain, then they were excluded from the multilevel logistic models (final n_Study1 = 87, n_Study2 = 160, n_Study3 = 257, n_study4 = 226). We also used multilevel linear models to assess the effect of facial muscle movement (“ExpressionActivation”) on intensity ratings on trials rated as painful in each study and to evaluate whether sociocultural measures impacted this relationship. For each multilevel model, we verified homoscedasticity of our residuals. We also report the full model formulas in the Supplementary Methods and model betas and significances for each of our predictors in the Supplementary Tables.