We are starting a series of blog posts interviewing different authors who have posted to PsyArXiv. The goal of these posts is to spotlight different uses and experiences of PsyArXiv and to help the community get to know some of our authors a little better. In our first post, Ben Brown, Associate Professor of Psychology at Georgia Gwinnett College and Chair of the PsyArXiv Steering Committee, interviews Cory Costello, a graduate student at the University of Oregon.
Ben: First of all, congratulations on your first first-authored paper and the attention that the preprint has received! As I mentioned previously, PsyArXiv is starting a series of blog posts interviewing different authors about their uses and experiences of PsyArXiv. To start, would you mind introducing yourself to our readers?
Cory: Thanks—both for the kind words and this opportunity! My name is Cory Costello, I’m a graduate student and doctoral candidate in the Psychology Department at the University of Oregon, working primarily with Sanjay Srivastava in the Personality and Social Dynamics Lab. Before getting to Oregon, I was an MA student in the Department of Psychology at Wake Forest University, where I worked primarily with Dustin Wood (now at University of Alabama). I did my undergrad at a small, public, Liberal Arts College called New College of Florida. My research interests revolve around personality assessment/measurement, personality development, interpersonal perception, reputation, and how personality is expressed and perceived online (in online social networking environments, such as Twitter). More broadly, I’m interested in methodology, data analysis, and how to make psychological science better, more open, and more reproducible.
Ben: Can you tell me a little about this new method for assessing personality?
Cory: The basic idea of the method was to take this idea of revealed preferences and apply it to personality assessment. Revealed preference techniques are used in some areas of psychology (e.g., close relationships/attraction) and seem to be even more common in behavioral economics. The basic idea of revealed preferences is that rather than asking people about their preferences for different features (e.g., do you find confident people attractive?), you can instead infer (or reveal) their preferences by correlating some criterion (a rating, a choice behavior, etc.) with the absence or presence of those features (e.g., do they rate more confident people as being more attractive?).
We attempted to take a similar approach to personality assessment, with the assumption that personality traits primarily describe patterns of trait-relevant behavior (e.g., someone’s level of a trait like dependable means they tend to more often behave in ways that are described as dependable). So we administered a bunch of short action scenarios to participants that set up a situation (e.g., you come home and your living space is messy from your roommate cooking) and asked them how likely they would be to respond in a particular way (e.g., how likely would you be to clean up the mess?). Then, we had another group of participants rate each of these actions (e.g., cleaning up after your messy roommate) on 23 bi-polar personality adjectives (e.g., dependable/reliable vs. undependable/unreliable). The correlation between the likelihood ratings and the independently rated trait-ness (we call this action characterization in the paper) of each action is what we’re calling a revealed trait estimate. So it’s sort of like one of those old choose your own adventure novels, but where your choices are being rated, and seen as reflecting your personality.
One kind of neat thing about this approach is that its pretty flexible. In addition to having the action scenarios rated for their trait-ness, we also had them coded for expected effects (e.g., the likelihood of being rejected), which we were interested in because of their relevance to personality trait judgments (which we had been working on in a previous, related project: Wood, Tov, & Costello, 2015). But in theory, you could code actions for other variables of interest.
I guess the last thing to mention is that we think this technique might be especially useful in contexts where more typical personality assessment may have problems. In our case, we were interested in applying it cross-culturally, as there have been some surprising results in cross-cultural comparisons of personality that may stem from measurement issues. Of course, some of the measurement issues may still affect revealed traits, but some might be mitigated. In particular, having a standard set of raters characterize the actions (rate their trait-ness) means that we can rule out that any cultural differences stem from using a trait term differently. For example, we can rule out that differences in revealed outgoingness between the two groups we looked at (which we found in both studies) are due to different understandings of what it means for an action to be outgoing, because that judgement is being made by a standard group. A similar technique (Situation Judgment Tasks) has been developed in the I/O psych literature on assessment to deal with some similar challenges in workplace assessment.
Ben: Sounds like a cool, novel approach! What do you think the benefits are of sharing such a novel approach on PsyArXiv?
Cory: My main motivation for sharing this particular work on PsyArXiv was to provide easier access to it. I first started working on this manuscript the summer of 2014. I had just finished my Master’s degree at Wake Forest and I hadn’t yet started the PhD program at University of Oregon. There was this 3 month gap in my access to all of the tools/resources provided by a university (library subscriptions, SPSS site license, etc.), but I was still trying to get work done on a few projects, including this one. Whenever I was searching for literature I was always so grateful to see a link to a PDF on Google Scholar (usually posted on a personal or lab website, since this was before PsyArXiv). I ended up with this list of articles I had to look up when I got access to a library subscription. I mean, it ended up working out alright in the end, but I can’t help but feel like a stronger preprint culture in psychology would have saved me a bit of a headache.
Incidentally, that summer is also when I first started learning R, which was prompted by not having access to SPSS. Necessity is a great motivator.
Had PsyArXiv existed when I first submitted this paper, I would have loved to post it earlier in the pipeline (either pre-submission, or at the time of initial submission) to elicit feedback. Sanjay Srivastava, Gerard Saucier, and I did that with a different paper, and received some interesting and helpful feedback on Twitter.
Ben: It sounds like that’s a pretty stable (and awesome!) motivating factor. How likely are you to share work on PsyArXiv in the future? Why?
Cory: I hesitate to say 100% likely—I have a pretty unshakable sense of uncertainty about just about everything. With that said, I’d say I’m extremely likely (effectively 100% likely) to share my work on PsyArXiv in the future. I’d say my primary motivation is still making the manuscript easier to access. As I alluded to in my last email, I’d like to get more in the habit of sharing pre-submission/working papers on PsyArXiv to elicit feedback, and so I suspect my motivation to share work on PsyArXiv will be increasingly driven by seeking feedback. I’m working on a few different manuscripts right now, and I think we (Sanjay and I) will post pre-prints at the time of initial submission (maybe before that; he and I haven’t explicitly discussed that yet).
One day, I’d like to post them before submission to potentially get feedback that I can incorporate into the manuscript before the initial submission, but I just don’t know if that extended timeline makes sense at my current career stage. In my perfect world, hiring committee expectations would be calibrated to a more extended timeline like that (e.g., preprints on a CV would be seen more positively; grad students would be expected to have fewer pubs, more preprints; etc.), but I don’t know if the field is there yet. So, for the time being, my goal is to post a preprint on PsyArXiv concurrently to submitting it to a journal, but hopefully I’ll be able to move posting the preprint to an earlier point in that pipeline eventually.
Ben: Thanks! What has been your experience of sharing your work on PsyArXiv? Anything surprising?
Cory: It’s been a positive experience. I’ll admit, it was nerve-racking when we posted the first preprint I was involved in (Sanjay posted the preprint), especially because we posted that at the time of initial submission. I had this sort of slight feeling of dread: ‘what if there was some obvious mistake in the manuscript or data/code we posted?! At least if these things come up in peer review it’s private!’ But, in the end, it was fine, and led to some really great feedback on Twitter.
I think the most surprising thing occurred with that same preprint I just mentioned. I was telling someone about the project recently, and they responded with, “Oh, this is the preprint you guys posted in PsyArXiv, right?” I guess I still sort of have in my mind that if a paper hasn’t been published in a journal, no one would know or care about it. But that preprint has been downloaded quite a bit (314 as of today). We’re still figuring out what to do with that manuscript (it was rejected on our initial submission), but I feel really grateful that the work is available to others in the meantime.
Ben: That’s encouraging to hear that PsyArXiv has helped to make your work more visible. Are there any new features you would like to see from PsyArXiv in the future?
Cory: I’d like to see features that sort of perform the curating role that journals often perform. It might be helpful if it recommended similar preprints (to the one you’re viewing). One thing that I think would be cool, and maybe the blog is aimed at doing this, would be to have some sort of monthly list of preprints (similar to subscribing to a journal). I’m thinking some sort of most viewed/downloaded set of preprints within topic area(s) the person has indicated interest.
Ben: Thanks for those ideas, Cory! I’ll pass them along and see what we can do. One last question: What would you say to a friend who is thinking about posting?
Cory: I would say go for it! It’s a great way to get your work out there – both so that people learn what you found, and so they can tell you how the work can be improved. Plus, it’ll make it easier for folks to find your work if they’re in between universities and still trying to get their research done.