Early Results of Process Quality Survey in Collaborative Networks

This summer I put out a call for people engaged in collaborative networks to take a survey about their experiences in those networks. Thank you to everyone who responded and who encouraged others to do the same. Although I may not be able to use the results in this round of research, you can still take an abbreviated version of  the survey if you like.

Below you can see some of the demographics of who responded to the survey. You will see that the answers don't always add up perfectly, that's because most of the questions did not require a response and some people chose not to respond.

Survey respondents' demographic info

You probably notice a lot of the same things I did when looking at the demographics. For example, the high number of responses from women (which is particularly noteworthy since nearly half of respondents said men and women were present in roughly equal numbers in their network), and the small number of respondents who are part of a racial minority in their network (of those 12 only 2 identified as white). Also, if you are like me, you are developing theories about why the demographics shook out as they did.

In this post, I will share some preliminary analysis of the quantitative data that was gathered regarding fairness. My next post will examine the quantitative data on collaboration. Future posts will explore open-ended responses, as well as dig more deeply into some of the interesting patterns that emerge.

Some words of caution: In general, a sample size of 30 or more is considered statistically valid. However, as you can see, we only barely have a statistically significant sample size of men and we do not have a statistically significant sample size for people who are in a racial minority in their network, nor for those in the governmental sectors. Moreover, those of us working in this field know that sometimes the term "network" is interpreted very differently by different people. I tried to filter for this by asking people entering the survey if they met certain criteria, but it's still hard to say how much "apple-to-apple" comparisons are really embedded in these results.

Lastly, because I want this blog to be readable, I am mainly avoiding detailed discussion of methodology, but please do reach out if you want more information about how I arrived at the interpretations below.

So, if you are willing to hold all these caveats and treat these results lightly, let's dig in!

Procedural Fairness 

The survey used the Process Quality Scale (PQS) developed by Darrin Hicks and colleagues to measure perceptions of how fair a process is (as opposed to how fair the outcomes of a process are, see my earlier blog post). This 15-question questionnaire deals with whether people see the process as authentic, revisable, and consistently applied. In the survey, I asked people to base their responses on their impressions about processes in their network overall, as opposed to one particular process.

The good news

In general, people find their networks to be engaged in processes that are mainly fair. At least, insofar as the PQS can measure such a thing. In 8 of the questions, the most common response was that processes are more fair than not, and in the other 7 questions, the most common response was that the processes are fair. 

I went into this analysis curious to see if a strong difference would be noted between members and staff/consultants. However, the results were virtually indistinguishable. Members seems to think their networks are a pretty fair place to be, and so do the staff and consultants. 

The interesting news

You don't have to be a novelist to know that readers don't want to hear about all the happy things, readers want the dirt! Not you, of course, but perhaps the other readers of this blog. Well, it turns out that when we break the data down, we see that the intensity with which people perceive procedural fairness varies significantly. 

I was surprised to see that the group that seems to have the strongest sense of fairness in their networks was men who are in a network that is majority-woman. The second strongest perceptions of fairness come from those who are in a racial minority in their network, although this group had a wider range of opinion than the men who were in gender minorities. 

Bubble chart showing the modal average responses to the PQS, broken down by subgroups

Women and men are the two groups that have the most pronounced differences in interpretations of the level of fairness. In 13 of the questions on the PQS, the most common response from men was that the process was fair. For women, they said the process was fair in response to only 3 questions, and for 11 others they said it was only more fair than not. To one question, the most common response from women was that things were more unfair than not (the question was about whether everyone has equal opportunity to influence decisions). 

I suspect the relatively strong feelings of fairness among racial minorities may have something to do with the fact that 75% of that group are men, and of those about half are in a network comprised largely of women. A larger sample size might reveal significantly different answers. As it is, respondents who were a racial minority were still the only group to have more than one question generate a response that things were more unfair than not (one question was about giving some more than they deserve while shortchanging others, another was about some people's merits being taken for granted while others have to justify themselves). 

As you can see, there is a lot to unpack. So stay tuned, and, if you want to see all the questions, feel free to go to the abbreviated version of the survey here. 

Interpretation

As I move forward in the research, I will be examining the other data and engaging in interviews to attempt to understand the differences in particular between men's and women's perceptions. Is it possible that women are more accustomed to collaboration than men, and thus situations that seem incredibly collaborative to men are merely acceptably collaborative to women? Could men be obliviously on the receiving end of deferential treatment? Could women be consulted less, or expected to pick up more of the procedural work? Is the fact that it was more challenging to get men to take the survey somehow related?  Most importantly, is it a problem that there is a discrepancy in the answers of men and women, and can this data help us make networks better? Drop me a line if you want to be part of the conversation!