Update September, 2018: We have launched new versions of our R scripts and supplements in preparation for a journal submission, in addition to migrating to a new home on the Open Science Framework (OSF) for this project.
All files and supplements can be found at https://osf.io/ntd97/
Our preprint, which discusses the survey-software method in detail and presents three validation studies, can be found at https://psyarxiv.com/hgy3z/. Please read this manuscript in detail prior to using iatgen.
iatgen is for noncommercial (e.g., academic) use only and is licensed by a Creative Commons Attribution-NonCommercial 4.0 International License. Any commercial use of this tool is strictly prohibited.
Welcome to iatgen, a free, easy-to-use tool that implements the “survey-software IAT” method described in detail in the methodological paper by Carpenter et al. (2018).
Using this tool, you can quickly implement their method, building and analyzing fully functional Qualtrics surveys that contain counterbalanced, seven-block Implicit Association Tests (IATs; Greenwald, McGhee, & Schwartz, 1998). These IATs can include text and/or image stimuli and are completed using the keyboard.
This page contains the following information:
- Get Started
- About Survey-Software IATs and iatgen
- One iatgen, Two Formats: Shiny App vs. R Package
- Frequently Asked Questions (FAQs)
- Cite iatgen
- Contact Us
Want to use iatgen? Go for it!
- Check out a preprint copy of our manuscript here. This describes the methodology behind the “survey-software IAT” as well as several important considerations. As iatgen implements this method, users should read it prior to using iatgen.
- Iatgen can be accessed as an R package (source file and tutorial documentation on github) or a web-app (available via Shiny apps). Our R package includes built-in documentation on each function, as well (e.g., ?cleanIAT()). Details on these two methods are given below.
- Additional documentation, tutorials, and examples are available via or OSF page.
About Survey-Software IATs and Iatgen
Iatgen implements the “survey-software” IAT method. So what are survey-software IATs and how do they work?
Survey-software IATs are IATs (for an overview of the IAT, click here) constructed out of modified survey elements (e.g., in a Qualtrics survey). That is, they are run fully within your Qualtrics survey and are run by your Qualtrics survey.
How do they work? Survey elements are modified by adding code to them, converting converting text-entry Qualtrics questions into fully functional IAT blocks. This code, developed by us, faithfully implements the IAT procedures as described in the IAT literature. When pasted into a properly-configured survey template, this creates a seven-block interactive IAT that fully counterbalances left/right starting positions of targets and categories (see video below). This survey-software IAT method has been rigorously validated in both crowdsource (MTurk) and student populations and is described in detail in our methods paper.
Iatgen implements this procedure for you. Please note that method could be implemented without iatgen (e.g., if someone wanted to manually edit code, copy/paste into survey template). This might be desirable, for example, for individuals looking to improve our method. All files necessary to do this are on our OSF page.
However, researchers don’t want to edit code. This is where iategn comes in–it fully implements our procedure and does all the technical work in just a few seconds. All you need to do is configure the desired IAT in iatgen tool, and it edits, copies, and pastes the IAT code into a survey template and makes a fully functional IAT survey immediately available for download and use.
For more information, check out of Frequently Asked Questions (FAQs) below or read our methods manuscript.
One Iatgen, Two formats: Shiny App vs. R Package
Iatgen can be run as either an R package (i.e., building IATs and analyzing data within R) or as a Shiny web app (i.e., building IATs and analyzing data with our user interface).
The Shiny Web Applet does everything through your web browser (no code required).
Access. The Shiny application can be found here! https://applibs.shinyapps.io/iatui2/
Build. The Shiny application IAT builder options controlled via on-screen menus.
Analyze. The Shiny applet allows you to quickly score your IAT. Simply export your data from Qualtrics in a legacy export CSV format and iatgen will clean your data and make it available for download (D-score algorithm; Greenwald et al., 2003) as well as provide key diagnostics (drop count, error rate, timeout rate, internal consistency estimates). It also performs a one-sample t-test to quickly tell you whether your IAT D-scores significantly differ from zero (i.e., is there implicit bias/association present), has options to export data, and has advanced scoring options so users can experiment with different analysis options.
The R Package
The R package is installed on a user’s computer and functions as an add-on to the R statistical software. We recommend users who run the R version use RStudio to run iatgen. The R package is essentially the same but allows more flexibility in analysis (what R does best). Plus, some people just really love R.
Access. If you wish to use the R version, an installation script can be run to install iatgen from github: InstallationScript.R. Note that we assume users are familiar with how to install and use R packages.
- All files necessary (scripts, tutorials, and more) can be found on the materials component of our OSF page.
- A tutorial and vignette for the iatgen R package can be found on our GitHub at www.github.com/iatgen/iatgen
Build. To build an IAT using the R package, a user runs an IAT-building script specifying the desired IAT from within R. This creates a Qualtrics Survey File (*.QSF) is created in the user’s working directory containing the desired IAT. This can then be uploaded to Qualtrics and customized (e.g., add explicit measures, consent documentation, manipulations, survey logic, etc.). The R package contains the same build features included in the web interface but may be more desirable for people who work with code.
Analyze. The R package includes advanced analysis features such as data cleaning and analysis following Greenwald et al.’s (2003) procedures (although all parameters can be customized). We provide analysis scripts so most data analysis code is pre-written; the user need not modify our code but is welcome to do so (see our built-in help documentation for more information about analysis functions). For those who analyze data in R, it is simple to take IAT data directly into analysis for any purpose (factor analysis, regression, ANOVA, etc.).
Frequently Asked Questions (FAQs)
1. How does iatgen run IATs?
Iatgen does not run IATs. Qualtrics does! What iatgen does is implements the survey-software IAT method automatically. That is, it configures code based on your input, copies it into a Qualtrics survey template, and outputs a Qualtrics survey containing your desired IAT–all in just a few seconds. However, importantly, iatgen is not a ‘software tool’ for running the IAT. It is simply a method for implementing the survey-software IAT method. Qualtrics is the software tool that runs the IAT.
We recommend you be familiar with how survey-software IATs work by reading our methods manuscript.
2. How is reaction-time measured? Does internet speed matter?
3. How much can I customize my IAT?
Technically, you can customize anything by modifying our code (if you are code savvy). If you do make improvements, share them with us and we will make them available. All code is available via our OSF page. That’s probably not the answer you want, however.
Assuming you don’t want to mess with code, our iatgen tool can automatically customize several features for you.
Using iatgen, users can easily customize: the number of trials in each block, the color of labels (and stimuli, for words), the use of words or images as stimuli, how errors are handled (display brief error message [Greenwald et al., 1998] or require that errors be corrected [Greenwald et al., 2003]), the duration between trials in milliseconds, the duration of the error message in milliseconds (if correction is not required), and the display of a direction-reminder note during the task. All of this is built into iatgen–that is, our tool has options to configure these features for you.
4. Can I edit the directions?
Yes, but not automatically. Simply build your survey in Qualtrics, then edit the HTML on each question to make the directions say whatever you want. Be sure not to edit any important HTML, however.
4. Can I make a Single-Category IAT?
Our method could be adapted to develop a single-category IAT. However, we do not have materials or tools for doing this at the present time.
5. How do I score the IAT?
Our iatgen package also includes easy analysis scripts. With just a few commands (or the use of our web interface), users can score the IAT using the D-score algorithm (including popular variants), get estimates of internal consistency, extract drop rates and error rates, and more. These scripts are available on our OSF page.
6. How do survey-software IATs compare with commercial software?
Actually, quite well. For more information, read our methods manuscript.
7. Why only Qualtrics?
8. Can I include several IATs in one survey?
You can, but it’s a bit tricky. To do this, start by building each individual IAT as a separate survey. Upload them each into a separate Qualtrics survey by creating a new survey > Tools > Import/Export > Import Survey. Then, to add one IAT to another, go to Add Block > Important Questions From… > My Surveys (or wherever your survey is saved in Qualtrics) > Import Entire Survey. The blocks from that IAT should then be inserted. In the Survey Flow, you should be able to all individual IATs each represented by its own randomizer.
When it comes time to analyze the data, as long as the question names for each IAT match what was originally created by iatgen (e.g., Q1.RP1) then our tool should be able to score each of them. However, as they will have the same names, you will need to score each one separately (split your CSV file in two: run each separately through iatgen’s web app or R scripts).
Alternatively, you could import all the data in one file into R (if you work in R), and adjust the variable names in our R scripts (e.g., if you want one reproducible analysis script to score your entire study).
9. What are the “permutations”? What do the variable names mean?
In our documentation, we refer to targets as “Target A” and “Target B” and to categories as “Positive” / “Negative.” We presume Target A is evaluated more positively, so a compatible block would be one where Target A is paired with Positive and Target B is paired with Negative).
An IAT typically has the starting positions (left/right) counterbalanced for both targets and categories. This means, technically, there are four possible configurations of an IAT:
- Target A could start on the right, initially paired with with the positive category (compatible block comes first). We call this permutation “RP.”
- Target A could start on the right, initially paired with with the negative category (incompatible block comes first). We call this permutation “RN.”
- Target A could start on the left, initially paired with with the positive category (incompatible block comes first). We call this permutation “LP.”
- Target A could start on the left, initially paired with with the negative category (compatible block comes first). We call this permutation “LN.”
To counterbalance targets and categories in Qualtrics, it’s (sadly) necessary to build each of these separately (done automatically by iatgen!). One is randomly to each participant.
IAT question names in Qualtrics include a 3-digit code identifying the permutation from which it comes. The first two digits represent the permutation (e.g., RP) and the third digit represents the block number. Thus, “Q9 RN2” is the second block in the permutation where Target A starts on the right side, initially paired with negative (i.e., incompatible block first).
For more information, read our methods manuscript.
10. iatgen says my data are missing! Help?
To score your IAT, Iatgen looks for your variables under the variable names used in our template. These should be something like “Q1 RP1.” When building your survey, you may have changed these names (e.g., copying content from one survey to another). Thus, the iatgen tool cannot find your data because you have renamed it. There’s an easy solution, however (see below).
11. I renamed my variables by accident and now I can’t score my IAT. Help?
Fortunately, this is an easy problem to solve. Simply open your CSV file of data in Excel and replace the variable names for the IAT data (should be 28 columns: 7 blocks x 4 permutations) with the correct ones. How can you get the correct ones? Simply create a new IAT with iatgen, run it as is (e.g., once through with yourself as a participant), and export the data. This should have the correct names.
12. Why use survey-software IATs?
In today’s world, much research is run online using survey programs such as Qualtrics. Historically, you could not build one within a survey; they had to be outsourced. This causes a number of issues (read our methods manuscript) and there are many reasons why you should prefer a survey-software IAT.
We sought to design a method that runs IATs within software that researchers already have. This has many advantages:
- It is friendly for online research (e.g., MTurk)
- Building and analyzing IATs takes just a few minutes (using iatgen tool)
- The entire study (IAT, manipulations, explicit measures, etc.) runs within one survey, reducing attrition/error risk and increasing flexibility for randomization, etc.
- Researchers have total control over study design and implementation. The IAT can be positioned anywhere within the study (or placed randomly within the study), just as with any other survey element.
- Data management is easy. Iatgen automates data cleaning (following Greenwald et al., 2003) and exports data (D-scores) for analysis; alternatively, all steps can be done manually using our R package for data analysis.
- Iatgen offers built-in support for other analysis features (e.g., reliability estimation). Users can also submit to us additional analysis functions and we can host it on our Open Science Framework page. It may be included in future updates!
- Reporting IAT diagnostics is much easier than it has ever been, given the automated report features built into the analysis tools. We hope this improves transparency and quality of published reports.
- Because you can easily build and run your own IATs within minutes, we hope applied users will find it easier to build and run IATs on the fly.
12. Can you give me information about data security or GDPR?
There are two things to know about data security. First, remember that iatgen does not run your study; Qualtrics does. Therefore, if you have questions about the security of your IAT data (or any data in an online survey), you should direct those questions to Qualtrics. Our general approach is to be cautious with promising absolute confidentiality to anyone about anything online. In fact, some institutions require disclaimers about the possibility that data can be intercepted (e.g., when downloading your data from a tool such as Qualtrics; when participants submit data to Qualtrics, etc.). However, we are not Qualtrics, nor are we in a position to advise you on the details of your study or what documentation you should provide. We encourage you to check with Qualtrics and your institution about what their policies are for online data collection.
The tool that most people have questions about is our data analysis tool. If you have any concerns over data privacy, you are welcome to download our R code and run it locally on your machine. This prevents you from transmitting data to/from our “Shiny” web application via the internet.
However, you may want additional details about our Shiny web application (including questions about GDPR, what data is stored after you use the tool, etc.). This information is available from the company that manages these tools, RStudio, and can be found here: https://rstudio.com/about/rstudio-and-the-gdpr-what-you-need-to-know. Further questions about the data kept by Shiny web applications can be directed to RStudio.
Want to reference iatgen? Please cite our methods paper. Click here for citation information.
- Do you have a question about iatgen?
- Are there features you are looking for?
- Do you have code you would like to add?
- Have you used it for an accepted presentation, grant, or publication?
Non-Commericial Use Only
Please note that the iatgen R package is licensed only for non-commercial (e.g., academic) use under a Creative Commons (CC BY-NC 4.0) license.
No warranty is offered and we assume no liability of any kind for any consequences that may result from using iatgen. This tool can be modified and distributed with attribution to us, but cannot be used for commercial purposes. More details are given in the full text of the license.
Although we believe the IAT can be validly run via Qualtrics (e.g., as set up via iatgen) and the use of Qualtrics as an IAT tool has been validated by Carpenter et al. (2018), this procedure and its code is not provided or endorsed by the IAT’s creators, and all code for this project was generated by iatgen’s creators.
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Image Tom Carpenter ©2015 Devil’s Punchbowl. The Devil’s Punchbowl is a collapsed cave on the Oregon coast that allows viewers to peer into the normally hidden world of tidal caves. Like the IAT, the Devil’s Punchbowl gives observers a window into hidden processes, inspiring awe and mystery.