First Look at Quant Analysis: The CBE SNA Study
I’m excited to share the details of my presentation at the North American Social Networks conference from this Tuesday. It’s the first look at the quantitative component of my mixed methods study–there’s a lot more to come! (overview at cbesnastudy.com)
Importantly, this was at a social network analysis (SNA) conference, so there is technical SNA jargon. The focus is also not on competency-based education and telling the story of the network. I’ll be producing more consumer friendly and CBE oriented publications/presentations in the future. I simply wanted to get something out into the world after so long. 🙂
Below is the transcript of the presentation, and here’s the accompanying slide deck.
If you really want to see and hear me talk through things, you can also find a recording here, but note that I did clean up what I had to say in the transcript below, including clarifying and adding some important points.
Any and all feedback is welcome. This includes connections to people who might find this interesting, and/or suggested articles or research you think I might learn from.
Finally, I want to express my sincere gratitude to American Institutes for Research, Lumina Foundation, National Academy of Education, and The Spencer Foundation for their partnership, funding, and support. To the University of Kentucky for my education (especially my chair John Nash, PhD). And, last but certainly not least, huge thanks to the people at or members of C-BEN (Competency-Based Education Network). Without their participation and support, I wouldn’t be anywhere close to where I’m at (and there’s still a long road of writings thing up!)
Hello, my name is Bruce helped and I’m a PhD student at the University of Kentucky. This is my presentation for the North American Social Networks conference. My presentation is a first look at the quantitative results for my study of the Competency-Based Education Network (C-BEN), which is really about the scale-up or the spread of higher education innovations—how that happens—especially through network organizations.
So in terms of the agenda today:
- First I’m going to talk a little bit about the study.
- Then the mixed methods research design, in particular, the area of the study that I’m looking at today being the quantitative portion of the study.
- I’ll go over the quantitative survey data
- Then I’ll dive into the quantitative analysis, first with an overview, and a summary in terms of networked findings,
- Then to the network and subgroups,
- Organization level analyses,
- Individual level analyses,
- A quick deep dive into the collaborative ties and some different attributes of those,
- Then my plans to extend the analysis
So about the study, again, this is a study of the Competency-Based Education Network. I’m exploring social capital and network leadership within this study of C-BEN.
CBE is short for competency-based education, which really started in the 1970s. CBE is something that’s been around for a long time. That said, competency-based education in its current iteration—its current conceptualization—and as it’s understood in this presentation, really started in the 2010, 2011, 2012 timeframe. And, the Competency-Based Education Network organization really came to life between 2013 and 2015.
So, in my study, I’m looking at how the formation of this membership organization of different higher education institutions and other organizations, how the relationships that were formed within that group and the activities of the network organization… how that’s influenced competency based education practices and how they’ve been implemented in different institutions.
Mixed Methods Research Design
In terms of the mixed methods research design, there is a quantitative and qualitative phase, as well as the mixed methods portion of this.
We start with quantitative phase and that’s where you can see that I’m integrating data from a lot of different places. The primary two sources, though, being the CBE Social Network Survey (CBESNS), which you see there on the bottom left of the visualization. That’s a survey that I constructed for this study to really get at relationships that exist among different participants in this network. And then also the American Institutes for Research and their National Survey of Postsecondary Competency Based Education (NSPCBE).
Those are the two main sources and, at least in this quantitative portion, it’s really mixing that data so I can look at how relationships are associated to the actual CBE program characteristics that you see, and programs that are implementing CBE so to see if there is actually an influence of social capital on adopted CBE practices.
Quantitative Survey Data
In terms of the survey data, and this is really for the CBESNS that I ran and implemented. There is the overall response rate for the 326 people that I invited to participate, and I received just under a 35% response rate for all those people that were invited.
I think a couple of important notes first.
85% of the 20 most central individuals participated. So all the people that responded to the survey, they identified people who are important and influential to them, and of all the people that tend to get nominated, the top of the top 20 of those individuals… most of them, 85%, have actually participated in this study.
Also, if you look at key organizations, and that’s defined as the more involved organizations, those that are participating on the boards, the committees, those that make a lot of presentations at the conferences. 72% of those organizations had someone that was actually participating in this study.
So, that’s who is providing data for the analysis. Now the response rate implications.
If you don’t have a higher overall response rate you can’t—it’s limited what you can do in terms of some of the whole network analysis procedures, or at least you really need to make sure that you’re really careful about what conclusions you’re making from certain analyses with some of those whole network analysis procedures where you might have to take some of them with a grain of salt.
One of the things I’ve done to shore up the analysis, the procedures, and the validity of the study, is that when I’m using whole network analysis, such as running QAP regression procedures, I’m limiting to the network only those people who actually responded to the survey. Then, there’s also an increased focus on ego network analysis [as well as the qualitative component of the study where I increased the number of interviewees from the planned 15-20 to actually interviewing 36].
Quantitative Analysis Overview
So, in terms of an overview, and again, looking at the network and subgroups, organization level analyses, individual level analyses, and the collaborative ties deep dive.
I think one important point to mention is that, at least with the individual level analyses, there’s actually two analyses.
In with my organization level analyses, I’m using logistic regression analysis. QAP logistic regressions (LR QAP) were performed within the UCINet social network analysis software.
With the individual level analyses, I did both the LR QAP regressions on individual collaborative relations, but then I also performed some ego network analysis, in particular with the homophily procedures [in UCINet], looking at how and whether shared characteristics among people were influential in their relationships.
Looking at the findings from across all of those different areas, and this was a bit of an experiment, to see if I could put all the findings from across all those different analyses into one slide.
Organization Level Analyses
So, what you see here on the left side, there’s three bubbles:
- Purple Shared CBE Practice, and
- Green Organizational Key Collaborative Relations,
- In blue, the Influential CBE Programs
Those are all the key dependent variables that were under investigation with the organization level analyses.
And so what I did see… what you can see is that, with the Shared CBE Practices, I did not find a statistically significant relationship to any of the other variables that I had.
That was one of the important questions that I had with the study. Related to the first research question for the overall study was, is there a relationship between social capital—the key collaborative relationships—and the eventual CBE design characteristics that you have there.
Now in the data that I do have right now, there was not a relationship, but I think there are a lot of other ways that you could imagine, with different data that there could still be a relationship. You just might need to conceptualize CBE Practices at a higher level, more granular lower level, or to look at only a specific characteristics [which I did as well, but again, maybe the definition wasn’t right to detect a direct relationship.
But at the same time, I think it’s also possible, certainly, to see how maybe, that maybe, there isn’t really a direct relationship between some of these specific design characteristics and the relationships that you have inside of an organizational network like this.
So, there’s other things that you’re really getting out of participation in a network—or a movement—like this now. With the other two variables, there were quite a lot of strong relationships, and so one of them, certainly it’s between organizational key collaborative relations and influential CBE programs.
There is definitely a relationship detected on both sides [whichever way you run the regressions]. So what’s driving that? Do key collaborative relations, do those potentially cause or lead to who you see as being influential programs? Maybe it could go that way, maybe it could be that you seek out key collaborative relations with people from influential CTE programs? Or a little bit of both, so that was an interesting relationship.
Also, looking at some of these other relationships in the gray boxes that are informing some of these variables.
Network and Subgroups
Now a couple of other things here, of course, on network subgroups. So on the next slide you’ll see the overall network, the view of the network.
Based on the key collaborative relationships that you see across organizations and when you look at subgroups using different community detection algorithms, what I saw in that was that there were a few things that were associated with the formation of some of those subgroups.
First of all was just the core actors in the network, the people that are really central. Those tend to be people who have been involved since the founding of this network starting back in 2012, 2013, all the way up to like the 2015-16 timeframe.
The other things that seemed to be driving sub groups are both the state location, in particular when you look at like at where, organized at the state level there was a sub CBE network, that might exist in some states.
That [state location] clearly seems to be driving formation of some of these subgroups.
Also, the theme or discipline. So, especially in one specific instance, you have a discipline or thematic area, which seems to be driving a lot of relationships between organizations and institutions.
Then, lastly, with a core, you know, when you talk about there being a core of different individuals, there is also a periphery, and so the periphery doesn’t tend to be connected.
But you do of course see that within a core periphery structure, you see the periphery and then also those other subgroups [based on state or theme].
Individual Level Analyses
Now the last two things and looking at the individual key collaborative relationships, when you look at the specific individuals and who they’re connected to. One of the big things you get out of doing this is that you can look at whether or not demographic and other variables are informing whether or not people tend to be more likely to be connected and working with one another.
So, in doing this, I was able to see that, no it’s not really demographics [variables related to gender, age, and race/ethnicity]. At least not informing things in this case. I was also able to see with the two analyses that I conducted that there’s also some mixed evidence in terms of the core actors and whether or not they inform whether or not relationships start. The network subgroups are definitely clear drivers of relationships. The state location, mixed opinions on that as well. Then also important were common ties and in degree centrality.
So that was individual key collaborative relations.
Collaborative Ties Deep Dive
Then, finally, this question about looking at the dimensions of key collaborative relations. So this is looking inside of these key collaborative relationships. What I would say is that it [the CBESNS questions] was asking individuals to look at their key collaborators, and to rate them, essentially, on the extent to which: An individual is someone that they would go to for knowledge or resources. That’s the knowledge resource access (KRA) variable.
Also, the level of trust they saw within their relationship. Trust instrumental being the extent that you trust that person to deliver on their commitment to you. More of a work related sort of a trust. Then trust expressive, which is more of a social-emotional-feeling type of a trust. That’s looking at the extent to which you might trust someone, that you might confide in someone. Share more confidential information. Such as whether or not you were considering another job or career change.
So those are three key variables and they’re key variables that seemed to drive [collaborative] relations in a lot of the theoretical literature on inter-organizational networks, and just networks in general.
So I wanted to look at that, and what you can see is that they certainly all have relationships with one another. In particular, you see strong relationships between knowledge resource access and instrumental trust, and also instrumental trust to expressive trust.
Some other things that seem to be correlated with those are having Met via C-BEN, is correlated with knowledge and resource access. Having a relationship with someone because you’re looking for knowledge, resources, and also instrumental trust, well that makes sense [in terms of the logic of why someone might have gone to C-BEN to begin with].
And if you’re looking at building trust within relationships [which is seen as incredibly important to an innovative and collaborative environment, and to assisting others in their implementation of change, particularly complex, tacit-knowledge based change], the fact that people had been involved in CBE collaborative work projects with one another, was strongly correlated with both instrumental and expressive trust.
Then, not surprisingly, expressive trust was more correlated with people that you participate in other activities with outside of CBE [NonCBE activities and collaborative projects]. Also with people that you were previously a co-worker with also correlated . Finally, strong levels of expressive trust correlates with people that you might consider to be your friends.
So, this was my attempt at networked findings. Now just quickly running through what’s building all that up.
So in the network and subgroups you can see, this is the overall network visualized based on key collaborative relationships. The nodes are sized by betweenness score, and probably not surprisingly, the network organization is the giant green node, or actor, that’s at the center of the core organizations, that just seem to be more connected.
Generally related to this, I would say with the founding of the organization, it’s a large group of these actors, and including some of the actors that are in some of these other colored subgroups. In particular, though, the groups that are in the middle, like the yellow and the green colored nodes. You also see here the two States sub networks that are very clearly identified, and then the theme or discipline oriented network as well.
Organization Level Analysis
In terms of the organizational level analysis, we already talked through some of that, but, again, the two key dependent variables, of the three that I tested are shown here.
No relationship with shared CBE practices that I found. I did find a lot more bivariate relationships [shown on slides in the appendix], but these are the two multiple regressions with key collaborative relationships and influential CBE programs. These are the variables that remained significant when adding multiple variables together in multiple regression procedures.
Individual Level Analysis
In terms of individual level analysis, the same thing so here was the most significant of the different multiple regressions. Here I’ve just summarized at the bottom the influential variables, which are the ones that are above, and things that were not influential like demographics, like gender or race/ethnicity. Someone’s age [the difference in individuals’ ages—does being close in age predict a relationship], didn’t seem to be driving relationships. Job type, and involvement in the founding the network, which is one of those points of mixed evidence.
Next, in terms of ego network analysis and looking at homophily [a quantitative preference for others who are similar]. Again, this is the likelihood that people’s attributes that are shared with someone else leads them to be more likely to be connected.
So here you see that with C-BEN founding involvement, here is a much stronger relationship. It’s the strongest of the ones that I had tested here actually. And that’s one where you see mixed evidence, because on the last slide with the multiple regression, you didn’t see that as being statistically significant, and yet you do see it here.
The other thing [in terms of mixed evidence] was that state location had appeared to be significant on the last slide (and also it appeared significant in the organization level analyses). But what I saw when, after I saw these results [indicating that State location doesn’t drive relations in general], I then drilled down into the data I saw that the State sub networks, a couple states and their networks, but one in particular, actually skew the overall data. This is because most of their members were just densely connected to one another and not to others [outside their State / subgroup]
So, certainly, this is something pointing to the fact that I need to continue with the quantitative analyses, and look much more specifically at each of the sub networks that are there.
Collaborative Ties Deep Dive
Lastly, understanding collaborative ties, and so this is where I ran QAP Correlation Analysis within UCINet. I was looking at knowledge and resource access and instrumental and expressive trust to see how those different things related. To talk through some of that, you can see some of those relationships here:
Circled in yellow, that is, the correlations between knowledge and resource access and then instrumental and expressive trust.
You can also see how that differs between moderate ties, between moderate plus ties [moderate-plus ties include all relationships where an alter was rated as somewhat useful or very useful for KRA, or somewhat or very trusting for instrumental/expressive trust; strong ties include only the relationships where an alter was rated very useful for KRA, or as having a very trusting relationship].
So you can see those connections there.
In blue you can see where collaborative work as the column header has that strong correlation to instrumental and expressive forms of trust.
With having Met via C-BEN in orange you can see where—and not with strong ties, but with just moderate plus (overall with looking at these ties)—that having Met through C-BEN had a correlation with instrumental trust and knowledge and resource access. So if you had a much stronger connection, maybe it was due to a more long standing relationship that might have come from somewhere else [versus a C-BEN connection], so there wasn’t as strong a correlation there.
And then last you see with expressive trust ties the connection, the correlation, with having been a prior coworker, and being involved in non-CBE activities, or collaborative work outside of CBE. Then also you see that strong expressive ties are correlated to friendship.
Plans to Extend the Analysis
So, plans to extend the analysis, where to go from here? So, to the extent that I can do all of this within the dissertation still, that’d be fantastic.
Some of this will likely need to be future research pursued afterward. Some of this will certainly be coming soon, though.
- Looking more deeply at C-BEN’s role in the network
- Looking at the relations within sub groups I’ve mentioned
- With the organizational analysis, rerunning it but excluding the NSPCBE outcomes data
Since I saw that there wasn’t a link between the outcomes and relations. This is because including the NSPCBE outcomes data restricted my dataset. It required me to look at just the intersection of survey responses from individuals and institutions that had responded to both the CBESNS and the NSPCBE. So, if looking just at the key collaborative relationships [only CBESNS responses], I would have a larger data set which could certainly change the analysis.
- Structural holes in collaborative and influencer ties, would like to look at that more with ego network analysis
- Deeper dive on collaborative work
So I have a lot more data on that than just, you know, yes or no to having past collaborative work. So, seeing how that links up to strong knowledge and resource access and trust.
- Looking at multi-embeddedness
So this is where you have these different people that have relationships that are CBE related, or within C-BEN, but some of these people also might have relationships that are outside of C-BEN, or outside of a CBE context and seeing how those different types of relationships—those different contexts—might influence things.
- Lastly, network robustness and resilience.
This is looking at what-if analyses, where if you were to remove specific nodes randomly, or very purposefully, to pull different people out, and to see what are the effects on the network.
So, those are some of the different plans to extend the analysis, at least from a quantitative perspective. There is also the whole qualitative portion of the study as well.
So with that I’d like to thank you for your attention, for your interest.
My contact information is here, including my email address. Also a link to my research project website where you can find an overview of the study at http://www.cbesnastudy.com. That’s where I will put the findings and visuals and such later on, as I finish up.
If there’s anyone that you know that you think would find this study interesting, certainly feel free to highlight them to me, or me to them, [or to share a link to this now]. I’d love to be in touch.
Also, if there’s any research that you’ve come across that you think I could learn and benefit from, I would also appreciate it if you could share that.
So, all that being said, that’s the presentation and I’m looking forward to sharing more as I continue through the study, finish the qualitative phases, etc. Thank you!
Gratitude to my dissertation committee and PhD program
- John Nash, PhD (Chair)
- Beth Rous, PhD
- Lars Bjork, PhD
- Ajay Mehra, PhD
Disclaimer (and appreciation!)
•This project is part of the National Research Collaborative for Competency Based Education/Learning, organized by the American Institutes for Research (AIR).
•This research is supported by grant funding from the Lumina Foundation, and the National Academy of Education / Spencer Foundation Dissertation Fellowship.