Survey-Based Implicit Association Tests: About and FAQs

Update September, 2018: We have launched new versions of our R scripts and supplements in preparation for a journal submission, in addition to migrating to a new home on the Open Science Framework (OSF) for this project.

All files and supplements can be found at https://osf.io/ntd97/

Our preprint, which discusses the survey-based method in detail and presents three validation studies, can be found at https://psyarxiv.com/hgy3z/. Please read this manuscript in detail prior to using iatgen.

iatgen is for noncommercial (e.g., academic) use only and is licensed by a Creative Commons Attribution-NonCommercial 4.0 International LicenseAny commercial use of this tool is strictly prohibited.

Welcome to iatgen, a free, easy-to-use tool that implements the “survey-based IAT” method described in detail in the methodological paper by Carpenter et al. (2018).

Using this tool, you can quickly implement their method, building and analyzing fully functional Qualtrics surveys that contain counterbalanced, seven-block Implicit Association Tests (IATs; Greenwald, McGhee, & Schwartz, 1998). These IATs can include text and/or image stimuli and are completed using the keyboard.

The iatgen tool, created as a supplement to the methods paper, implements the entire procedure automatically (no need to edit code!). Users simply customize their IAT with the iatgen tool (web application or downloadable R package), and the software builds a ready-to-run Qualtrics Survey File (QSF) containing the IAT. The survey can then be customized as desired (e.g., explicit measures, randomization, additional IATs, etc.). Alternatively, researchers can use iatgen to customize existing HTML/JavaScript code and then copy/paste it into a Qualtrics survey or template. When research is complete, iatgen includes a full suite of analysis tools, performs data reduction, provides diagnostics (e.g., internal consistency, error rate, etc.), and exports clean data for analysis.

This page contains the following information:

  1. Get Started
  2. About Survey-Based IATs and iatgen
  3. One iatgen, Two Formats: Shiny App vs. R Package
  4. Frequently Asked Questions (FAQs)
  5. Cite iatgen
  6. Contact Us
  7. License

In addition, you can get more information about the IAT here, and view an annotated list of materials on our materials page (all linked to our OSF page).


Get Started

Want to use iatgen? Go for it!

  1. Check out a preprint copy of our manuscript here. This describes the methodology behind the “survey-based IAT” as well as several important considerations. As iatgen implements this method, users should read it prior to using iatgen.
  2. Iatgen can be accessed as an R package (source file and tutorial documentation on github) or a web-app (available via Shiny apps). Our R package includes built-in documentation on each function, as well (e.g., ?cleanIAT()). Details on these two methods are given below.
  3. Additional documentation, tutorials, and examples are available via or OSF page.

About Survey-Based IATs and Iatgen

Iatgen implements the “survey-based” IAT method. So what are survey-based IATs and how do they work?

Survey-based IATs are IATs (for an overview of the IAT, click here) constructed out of modified survey elements (e.g., in a Qualtrics survey). That is, they are run fully within your Qualtrics survey and are run by your Qualtrics survey.

How do they work? Survey elements are modified by adding code to them, converting converting text-entry Qualtrics questions into fully functional IAT blocks. This code, developed by us, faithfully implements the IAT procedures as described in the IAT literature. When pasted into a properly-configured survey template, this creates a seven-block interactive IAT that fully counterbalances left/right starting positions of targets and categories (see video below). This survey-based IAT method has been rigorously validated in both crowdsource (MTurk) and student populations and is described in detail in our methods paper.

Iatgen implements this procedure for you. Please note that  method could be implemented without iatgen (e.g., if someone wanted to manually edit code, copy/paste into survey template). This might be desirable, for example, for individuals looking to improve our method. All files necessary to do this are on our OSF page.

However, researchers  don’t want to edit code. This is where iategn comes in–it fully implements our procedure and does all the technical work in just a few seconds. All you need to do is configure the desired IAT in iatgen tool, and it edits, copies, and pastes the IAT code into a survey template and makes a fully functional IAT survey immediately available for download and use.

For more information, check out of Frequently Asked Questions (FAQs) below or read our methods manuscript.


One Iatgen, Two formats: Shiny App vs. R Package

Iatgen can be run as either an R package (i.e., building IATs and analyzing data within R) or as a Shiny web app (i.e., building IATs and analyzing data with our user interface).

Shiny

The Shiny Web Applet does everything through your web browser (no code required).

Access. The Shiny application can be found here! https://applibs.shinyapps.io/iatui2/

Build. The Shiny application IAT builder options controlled via on-screen menus.

Analyze. The Shiny applet allows you to quickly score your IAT. Simply export your data from Qualtrics in a legacy export CSV format and iatgen will clean your data and make it available for download (D-score algorithm; Greenwald et al., 2003) as well as provide key diagnostics (drop count, error rate, timeout rate, internal consistency estimates). It also performs a one-sample t-test to quickly tell you whether your IAT D-scores significantly differ from zero (i.e., is there implicit bias/association present), has options to export data, and has advanced scoring options so users can experiment with different analysis options.

The R Package

The R package is installed on a user’s computer and functions as an add-on to the R statistical software. We recommend users who run the R version use RStudio to run iatgen. The R package is essentially the same but allows more flexibility in analysis (what R does best). Plus, some people just really love R.

Access. If you wish to use the R version, an installation script can be run to install iatgen from github: InstallationScript.R. Note that we assume users are familiar with how to install and use R packages.

Build. To build an IAT using the R package, a user runs an IAT-building script specifying the desired IAT from within R.  This creates a Qualtrics Survey File (*.QSF) is created in the user’s working directory containing the desired IAT. This can then be uploaded to Qualtrics and customized (e.g., add explicit measures, consent documentation, manipulations, survey logic, etc.). The R package contains the same build features included in the web interface but may be more desirable for people who work with code.

Analyze. The R package includes advanced analysis features such as data cleaning and analysis following Greenwald et al.’s (2003) procedures (although all parameters can be customized). We provide analysis scripts so most data analysis code is pre-written; the user need not modify our code but is welcome to do so (see our built-in help documentation for more information about analysis functions). For those who analyze data in R, it is simple to take IAT data directly into analysis for any purpose (factor analysis, regression, ANOVA, etc.).


Frequently Asked Questions (FAQs)

1. How does iatgen run IATs?

Iatgen does not run IATs. Qualtrics does! What iatgen does is implements the survey-based IAT method automatically. That is, it configures code based on your input, copies it into a Qualtrics survey template, and outputs a Qualtrics survey containing your desired IAT–all in just a few seconds. However, importantly, iatgen is not a ‘software tool’ for running the IAT. It is simply a method for implementing the survey-based IAT method. Qualtrics is the software tool that runs the IAT.

We recommend you be familiar with how survey-based IATs work by reading our methods manuscript.

2. How is reaction-time measured? Does internet speed matter?

Because the survey-based method uses JavaScript, we instruct the code to download and pre-load all stimuli prior to displaying the IAT on the screen. In addition, we can instruct the software to only transmit data back and forth to Qualtrics between IAT blocks. The actual work of reaction-time capture does not require any data to be transmitted over the internet.

To capture reaction times, we use JavaScript code validated for psychological research (de Leeuw, 2015; Reimers & Stewart, 2015). For more information, read our methods manuscript.

3. How much can I customize my IAT?

Technically, you can customize anything by modifying our code (if you are code savvy). If you do make improvements, share them with us and we will make them available. All code is available via our OSF page. That’s probably not the answer you want, however.

Assuming you don’t want to mess with code, our iatgen tool can automatically customize several features for you.

Using iatgen, users can easily customize: the number of trials in each block, the color of labels (and stimuli, for words), the use of words or images as stimuli, how errors are handled (display brief error message [Greenwald et al., 1998] or require that errors be corrected [Greenwald et al., 2003]), the duration between trials in milliseconds, the duration of the error message in milliseconds (if correction is not required), and the display of a direction-reminder note during the task. All of this is built into iatgen–that is, our tool has options to configure these features for you. 

4. Can I edit the directions?

Yes, but not automatically. Simply build your survey in Qualtrics, then edit the HTML on each question to make the directions say whatever you want. Be sure not to edit any important HTML, however.

4. Can I make a Single-Category IAT?

Our method could be adapted to develop a single-category IAT. However, we do not have materials or tools for doing this at the present time.

5. How do I score the IAT?

Our iatgen package also includes easy analysis scripts. With just a few commands (or the use of our web interface), users can score the IAT using the D-score algorithm (including popular variants), get estimates of internal consistency, extract drop rates and error rates, and more. These scripts are available on our OSF page.

6. How do survey-based IATs compare with commercial software?

Actually, quite well. For more information, read our methods manuscript.

7. Why only Qualtrics?

We developed iatgen within Qualtrics because it is widely used and offers support for JavaScript and HTML editing, which is needed to make the IAT run. Conceptually, our code could run within any online survey tool that offers rudimentary JavaScript and HTML support so long as a few criteria are met. We look forward to expanding iatgen as other mainstream tools become more available.

8. Can I include several IATs in one survey?

You can, but it’s a bit tricky. To do this, start by building each as a separate survey. Upload them into Qualtrics. Then, copy the contents from one into the other and edit the survey flow as desired. Each IAT should be represented by its own randomizer.

When it comes time to analyze the data, as long as the question names for each IAT match what was originally created by iatgen (e.g., Q1.RP1) then our tool should be able to score each of them. However, as they will have the same names, you will need to score each one separately (split your CSV file in two: run each separately through iatgen’s web app or R scripts).

Alternatively, you could import all the data in one file into R (if you work in R), and adjust the variable names in our R scripts (e.g., if you want one reproducible analysis script to score your entire study).

9. What are the “permutations”? What do the variable names mean?

In our documentation, we refer to targets as “Target A”  and “Target B” and to categories as “Positive” / “Negative.” We presume Target A is evaluated more positively, so a compatible block would be one where Target A is paired with Positive and Target B is paired with Negative).

An IAT typically has the starting positions (left/right) counterbalanced for both targets and categories. This means, technically, there are four possible configurations of an IAT:

  • Target A could start on the right, initially paired with with the positive category (compatible block comes first). We call this permutation “RP.”
  • Target A could start on the right, initially paired with with the negative category (incompatible block comes first). We call this permutation “RN.”
  • Target A could start on the left, initially paired with with the positive category (incompatible block comes first). We call this permutation “LP.”
  • Target A could start on the left, initially paired with with the negative category (compatible block comes first). We call this permutation “LN.”

To counterbalance targets and categories in Qualtrics, it’s (sadly) necessary to build each of these separately (done automatically by iatgen!). One is randomly to each participant.

IAT question names in Qualtrics include a 3-digit code identifying the permutation from which it comes. The first two digits represent the permutation (e.g., RP) and the third digit represents the block number. Thus, “Q9 RN2” is the second block in the permutation where Target A starts on the right side, initially paired with negative (i.e., incompatible block first).

For more information, read our methods manuscript.

10. iatgen says my data are missing! Help?

To score your IAT, Iatgen looks for your variables under the variable names used in our template. These should be something like “Q1 RP1.” When building your survey, you may have changed these names (e.g., copying content from one survey to another). Thus, the iatgen tool cannot find your data because you have renamed it. There’s an easy solution, however (see below).

11. I renamed my variables by accident and now I can’t score my IAT. Help?

Fortunately, this is an easy problem to solve. Simply open your CSV file of data in Excel and replace the variable names for the IAT data (should be 28 columns: 7 blocks x 4 permutations) with the correct ones. How can you get the correct ones? Simply create a new IAT with iatgen, run it as is (e.g., once through with yourself as a participant), and export the data. This should have the correct names.

12. Why use survey-based IATs?

In today’s world, much research is run online using survey programs such as Qualtrics. Historically, you could not build one within a survey; they had to be outsourced. This causes a number of issues (read our methods manuscript) and there are many reasons why you should prefer a survey-based IAT.

We sought to design a method that runs IATs within software that researchers already have. This has many advantages:

  • It is friendly for online research (e.g., MTurk)
  • Building and analyzing IATs takes just a few minutes (using iatgen tool)
  • The entire study (IAT, manipulations, explicit measures, etc.) runs within one survey, reducing attrition/error risk and increasing flexibility for randomization, etc.
  • Researchers have total control over study design and implementation. The IAT can be positioned anywhere within the study (or placed randomly within the study), just as with any other survey element.
  • Data management is easy. Iatgen automates data cleaning (following Greenwald et al., 2003) and exports data (D-scores) for analysis; alternatively, all steps can be done manually using our R package for data analysis.
  • Iatgen offers built-in support for other analysis features (e.g., reliability estimation). Users can also submit to us additional analysis functions and we can host it on our Open Science Framework page. It may be included in future updates!
  • Reporting IAT diagnostics is much easier than it has ever been, given the automated report features built into the analysis tools. We hope this improves transparency and quality of published reports.
  • Because you can easily build and run your own IATs within minutes, we hope applied users will find it easier to build and run IATs on the fly.

Cite iatgen

Want to reference iatgen? Please cite our methods paper. Click here for citation information.


Contact Us

  • Do you have a question about iatgen?
  • Are there features you are looking for?
  • Do you have code you would like to add?
  • Have you used it for an accepted presentation, grant, or publication?

Click here to contact us.


Non-Commericial Use Only

Please note that the iatgen R package is licensed only for non-commercial (e.g., academic) use under a Creative Commons (CC BY-NC 4.0) license. 

No warranty is offered and we assume no liability of any kind for any consequences that may result from using iatgen. This tool can be modified and distributed with attribution to us, but cannot be used for commercial purposes. More details are given in the full text of the license.

Although we believe the IAT can be validly run via Qualtrics (e.g., as set up via iatgen) and the use of Qualtrics as an IAT tool has been validated by Carpenter et al. (2018), this procedure and its code is not provided or endorsed by the IAT’s creators, and all code for this project was generated by iatgen’s creators.

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.


Image Tom Carpenter ©2015 Devil’s Punchbowl. The Devil’s Punchbowl is a collapsed cave on the Oregon coast that allows viewers to peer into the normally hidden world of tidal caves. Like the IAT, the Devil’s Punchbowl gives observers a window into hidden processes, inspiring awe and mystery.

Advertisements