Earlier this month Phil asked a question on Twitter about the growing usage of (and pushback against) faculty training based on the Quality Matters Course Design Rubric. That question led to a rich discussion – both pro and con – on the usage of the QM rubric in the attempt to improve online teaching in Fall 2020. The QM staff requested that we help with an alternate forum for them to address some of the issues raised online.
This is the third in a special series of podcast episodes on an important topic as we try to migrate from emergency remote teaching to purposely-designed quality online education. Link to Jesse’s blog post on the subject.
- 15A: Introduction of topic
- 15B: Interview with Bethany Simunich and Brendy Boyd from Quality Matters
- 15C: Interviews with Stephanie Moore and Jesse Stommel
Podcast: Play in new window | Download
Subscribe: Apple Podcasts | Google Podcasts | RSS | More
Transcript:
Phil: Welcome to COVID Transitions, where we discuss the transition that higher education has gone through and is going through due to the Covid-19 pandemic. I’m Phil Hill, and in this episode, I interview Stephanie Moore and Jesse Stommel to get a deeper discussion on the critical perspective of how Quality Matters and its course design rubric are being used in schools, particularly the spring and summer.
I’m here with Stephanie Moore, recently of the University of Virginia, but on her way to a new post with the University of New Mexico. A collaborator: I got the chance to co-write an article with Stephanie early on about the Covid transition. So it’s great to meet with you in person. Actually, I think this is the most live that we’ve met before, so it’s good to virtually meet, Stephanie.
Stephanie: That’s right. Good to meet you, too.
Phil: So were you surprised to see how much commentary came out, and [00:01:00] what was your impression of it?
Stephanie: Yeah, I have to admit, I was, too. I mean, in some ways, I guess I should say yes and no. You know, I know how faculty feel about quality matters, and it’s really a mixed bag. I think most of the folks who I know, who I would describe as seasoned educators who have a very clear sense of what they like to do in their classroom, they know themselves as educators. They know what they want to do. Those tend to be the folks who are more frustrated with it and feel like it binds them more than it supports them. Whereas there are faculty, especially those who are very new to online and typically those who are really new instructors like newly hired teachers, they tend to like Quality Matters more, in part because they feel like it gives them ideas and scaffolding and tools that they’re [00:02:00] just not familiar with anymore. So I think you get a mix of reactions that hinges largely on people’s experience and their comfort level with instruction and especially their comfort level and experience with teaching online as well.
Phil: And as usual, as with your writing, you just packed a lot into that space. I wouldn’t mind unpacking a little bit. First of all: what is it? “Frustrated with it”, and what does that mean in terms of Quality Matters? Within Quality Matters, the rubric versus all of the services of Quality Matters, but then also Quality Matters versus how institutions are applying it. So can you get that down a little bit?
Stephanie: I think the best way to maybe tackle it is to talk about how we had a conversation about it occurring when I was there in the Curry School of Education. When we sat down, and this was pre-Covid [00:03:00] when we sat down, to have a conversation about what does quality or effective online learning really mean for us. Our faculty asked me to lead a half day workshop on this, where I went through: Well, here’s what the research has to say. And we looked at Quality Matters, OLC, some other examples out there. And I had developed one in-house as well, because a lot of times it’s these are proprietary – you have to pay to use them. And I had been asked in a previous setting to develop something so that we didn’t have to pay for that. And it’s all anchored in the same research-based principles and things like that. And it was interesting watching the faculty react to all of these different pieces. They really like the in-house model that I had developed, but they felt like that was more of an articulation, of a vision of what is it that we really [00:04:00] want online learning to mean, and to be for the Curry School of Education. And they liked the idea of using that as an anchor for annual evaluations as well, that they felt like the way that that was articulated was more aspirational and visionary. But yet when it came down to the nuts and bolts about, ‘OK, so I’ve got to sit down and actually build my course, how do I get that done?’ They liked various aspects of Quality Matters and OLC, in order to scaffold them, to help them get that done, but only insofar as it fit into that vision.
And I think that captures the tension that we see and feel really nicely, that a lot of people feel like, you know, the what Quality Matters or OLC or others help with is the nuts and bolts, or the micro level details of how to build a course. A lot of institutions [00:05:00] end up using the Quality Matters rubric as the definition of excellence, as the vision for online learning, and I think that’s where that tension point comes on. So and I think that’s what you’re seeing in the in the conversations that were happening on Twitter, too, is a lot of people were saying, the way in which my institution uses, it actually binds my ability to design quality online learning, and the way that I really want to go about that. And I know Quality Matters, they don’t intend it to be that way. It’s not really designed to be that way. And yet that’s that’s how it’s getting used by administrations. And so in that regard, it’s like any other tool in that when administration or leadership starts to use a tool in a certain way, associations get made [00:06:00] with that tool.
And and I do feel I really feel for Quality Matters because I feel like they’ve got like the rubric itself is nicely anchored in research based principles. But they struggle with a message that there’s a singular way to design an effective online course. And those of us who have been teaching know that whether it’s online or face to face are blended, there’s no singular way. There’s no singular type of course, there’s no particular way to go about this. And I know based on their conversations, or conversations with them, that that’s not what they intend, and that they really struggle with that message. But yet that’s the impression that a lot of faculty have, is that ‘this is a particular type of online course, and maybe Quality Matters is great for that, but I really want to build a very different kind of course.’ [00:07:00]
And frankly, we see the challenge comes in with courses when you’re working with adult learners, and you want to give your learners a lot more autonomy around how the course gets structured, setting class expectations, choosing what you’re going to cover in class, structuring and sequencing that. So in graduate school in particular, which is where a lot of online learning takes place, faculty express real tension between their educational philosophies or epistemology and what they feel the Quality Matters rubric is suggesting they do instructionally.
Phil: I’m hearing two things, and they’re not contradictory, but I want to make sure I’m hearing both. On one hand, a large amount of the pushback comes from the way administrators use Quality Matters and use it as almost a QA process and [00:08:00] not holistically. But at the same time, the Quality Matters rubric does lend itself to a particular course design, set of assumptions. So it does contribute to the design, this way type of mentality. So am I hearing that both of those are happening?
Stephanie: I think that’s that nicely summarizes both my own experience and what I felt I was hearing in the conversations on Twitter as well. There was one colleague who posted how Quality Matters tends to suggest a very constructivist approach. And honestly, that’s not a critique we hear just about Quality Matters, but about instructional design broadly. That instructional design tends to privilege the instructors’ decisions about how to structure everything rather [00:09:00] than being a more participatory process. And so it’s not terribly surprising to me to hear that. But I do think that the field of instructional design broadly has for some time been in process of pivoting away from that instructivist sort of perspective. I think the other challenge here, percolating in all of this, is that there’s a there’s a class issue in innovations, and diffusion of innovations, that I think is going on here, too. And that said, the developers are the creators of an innovation, invent it one way and envision it being used one way, and then it gets put out into society or whatever context, and it gets used in a very different way from how the designer or developer intended it to be used. And we see that gap between designer or developer [00:10:00] intent and actual implementation all the time.
And and I do think that Quality Matters, just like any other sort of innovative or entrepreneurial entity has to really think about, ‘OK, how much ownership are we going to take over the implementation that’s going on and and whether or not that maps to what our vision was and how that’s affecting the perceptions and the branding of our product that we’ve created’ versus how much they they don’t want to want to get involved in that. So I think they’ve got a very interesting conversation internally to have right now. You know, I’m not sure I have specific suggestions, but I have a few thoughts on how that gets managed. But, I do think that I would like to see them be reflective [00:11:00] about this feedback that they’re getting and really think about how can we be different partners? Can we be better partners not just with the administration’s or the institutions, but with the faculty who are really the ones where the rubber meets the road? And what’s happening at that point of contact is not always a very happy experience.
Phil: So even if it’s not something, that you said, that they intended.
Stephanie: Exactly.
Phil: Then taking a role in hearing where the frustration is and seeing how they might be able to influence the implementation, as opposed to just looking back, saying, ‘well, that’s not what we said to do.’
Stephanie: Yeah.
Phil: This is taking a little bit different way. But although QM, they’re much bigger than just the rubric, but there is a centrality around the course design process as opposed to getting into the course facilitation and teaching.
Stephanie: Right.
Phil: And the very name of Quality Matters. [00:12:00] And it has this implication of quality comes from course design. So when you were at Curry or just in general, what’s your view of what role should QM play outside of course design, or what role do you see them playing more into the teaching and facilitation?
Stephanie: Boy, that’s a great question, because, when we sat down, and we were actually cutting the rubric and different things apart, and reorganizing and moving it around, which is fascinating to watch, right, how faculty were thinking about it, interpreting these different pieces? And so, what we ended up with was the the Quality Matters pieces that faculty wanted to retain really did end up in the course development phase. There wasn’t [00:13:00] anything that they felt was about the course implementation phase of things. Now, there are some pieces in there that I think actually are, like timeliness of responsiveness on the part of faculty, things like that, that all come at the point of implementation.
I think even that you get into some tricky spaces where if Quality Matters were to decide, you know what, let’s flush out the implementation phase of a course and provide some guidance around that, having written guidance myself around this, it’s very easy to start to map out guidance that can make every course look like a cookie cutter. And so I think whether they do that, or just focus on the development piece alone, I really think they need to think about how do we communicate diversity of opportunities, or diversity [00:14:00] of design ideas, to instructors to get to put more of a focus on imagination, or just simply a range of different approaches as opposed to a good course. Is this a good course? Broadly, those of us who know the research would say there’s some broad principles that certainly crop up for an effective online course versus an ineffective course. But those principles don’t start to drive a particular sort of design, whereas when you start getting into the nuts and bolts about, make sure you do this and make sure you do that, that that’s when you start driving things in a particular direction for courses. I’m not sure I answered your question, Phil..
Phil: It’s a conversation, right? It’s not a Q&A. So that is helpful.
Stephanie: And and that all evolves as a result of a very social process [00:15:00] where, you know, innovations aren’t the they’re not owned by the developer, by the innovator. You know, the people adopting it have a lot to say back. Yes. And a lot of input back into that. And so I think once you understand that, it just it just sort of is how things develop.
And so I think for an entity like QM, the best thing that they could do would really be to look at something like that and say, OK, this is how it goes. And rather than trying to fight the social process, what if we actually adopt that as our way of doing things, and we’re collaborative and iterative along with the very people that we’re trying to work with? And none of that is to suggest that QM is not thinking that way or anything. I think what I heard in response from Brenda and Bethany, ‘we’re very positive, very engaged.’ And I think that’s a healthy way to go on the dialogue. [00:16:00]
So with faculty, we are used to having shared governance. And so we’re not simply answering to administration. We want to have a say in how things go because we feel like it’s it’s part of how universities are structured. Some universities will push on this more than others. But as far as faculty, we believe in having a shared say in the vision for what what it is that we are trying to do, and how we are trying to move that forward. If you’re communicating to that group, to faculty, that you’re not listening, if administrators feel like, ‘fine, whatever, this free tool,’ faculty feel like you’re not listening, you’re actually missing half of the governance structure of universities. And if you take a defensive posture to [00:17:00] that, you really do risk excluding a very influential voice and the decisions that get made in institutions of higher education.
Phil: Sure. And I would add to that just adding in my own view, you’re also likely to trigger even more extreme reactions. So it’s not just missing out, but it’s a difficult area.
Stephanie: Yeah, the analogies that I hear a lot of faculty use to QM are not flattering at all. And so if they’re already frustrated with the tool itself and the way in which it’s being implemented, and then they voice that and what they get back in return is defensiveness. And you’re all wrong at it. It’s simultaneously denying our experiences, which, as you can tell, our very shared experiences across institutions.
Phil: Yeah, I would say the mischaracterization risk goes both ways. Part of the risk is people [00:18:00] mischaracterizing what QM intentions are, what they provide. But at the same time, there’s a risk of them mischaracterizing the pushback they’re hearing.
Stephanie: I think that’s a good summary.
Phil: But I really appreciate your time taking on this and hopefully this in a different modality will be useful. So I appreciate your your help on this.
Stephanie: Thank you. Phil, good to talk.
Next up is my interview with Jesse Stommel.
But Jesse, welcome. And if you could give the listeners, you know, let them understand where you’re coming from.
Jesse: Great. Great to talk to you. So I’m Jesse Stommel. I have been teaching for a little over 20 years, and my research focuses in higher education pedagogy, and specifically critical digital pedagogy. I am the executive director of Hybrid Pedagogy and an associate director of Digital Pedagogy Lab, and [00:19:00] it’s great to join you. I’m looking forward to this conversation.
Phil: Well, thanks. Well, to jump into it, this is clearly a topic that you’ve been thinking about. In other words, you were not just reacting to a Twitter conversation. And as a matter of fact, you’ve written a blog post that I believe was associated with a with a talk that you were giving. But to jump into it, what started the whole conversation was that I was seeing a lot more pushback on the usage of Quality Matters in terms of schools that are trying to help their faculty move into more of a quality online approach. Has this been a topic that you’ve been following and looking out for a while, or is this a fairly new interest of yours?
Jesse: Conversations about Quality Matters are something that I’ve been a part of for over a decade, and in different ways, at different levels. I’ve been at institutions [00:20:00] that have adopted the Quality Matters rubric. I’ve given presentations where I’ve talked about and analyzed the Quality Matters rubric, and I’ve really worked on this from all different sides. I’ve been an instructional designer. I’ve been an online instructor. I’ve been a face to face instructor. I’ve been an administrator. And so I’ve really given a lot of thought to the Quality Matters rubric and how it’s used at institutions. I’ll just be really honest and straightforward and say that I’ve never been a big fan of the Quality Matters rubric. Doesn’t mean that I don’t think there are some really great people working at Quality Matters, and working to help faculty move online. And it also doesn’t mean that I don’t think that there are some wonderful faculty members and administrators who are using the Quality Matters rubric and effective ways. I’ll tell you that the germ of this conversation more recently for me was watching how institutions were employing and rolling out the Quality Matters rubric in response to the pandemic pivot [00:21:00] to online learning.
Yes. And ultimately, my concerns about Quality Matters got a lot greater in this moment because I just don’t feel like it is a good tool to help brand new online teachers respond to and engage in emergency remote learning. I feel like the tool, and we can get into this, but I feel like the tool is far too elaborate. And when I see faculty grappling with this tool in the midst of an emergent crisis, what I see is a lot of faculty feeling like they have no way forward. Like there’s no point of entry for them into this conversation about online learning and that the Quality Matters rubric frustrates that even further.
Phil: And just to provide some context, because I know it’s a complicated subject – on Twitter part of the thing was about the rubric, and then there was also the [00:22:00] issue of the broader professional development of Quality Matters – but then we get into how schools are choosing to use it, sort of quasi independent of Quality Matters. So could you describe sort of the context? Are you talking about the rubric itself or are you talking about how schools are applying it, which might not even be what Quality Matters intended? Or is it a combination?
Jesse: I think it’s a combination of both. If I look at the rubric itself as an instrument, we can have a whole conversation about the issues that I take with it. But I think that the bigger problem right now is the way that institutions are are using it. And ultimately, the biggest problem is that is that I see a lot of institutions using this in an obligatory way, where essentially what it is is not a tool to help people become better online teachers, but is a tool for quality assurance – that it is a mechanism that institutions [00:23:00] are using to certify that their online teaching is good, and they’re basically running their faculty. And these are faculty who have never even thought, in some cases never even thought about teaching online. And these faculty are being run through what is essentially a bureaucratic juggernaut for them, a process that feels utterly at odds with who they are as teachers and what they feel like their work in education is. And so ultimately, the problem is the making of rubric like this, a 42 or 43 point rubric, very hyper precise, making it obligatory and also not really dropping faculty into a process of conversation around it where they feel like they can consent and be full participants in the work that this rubric inspires.
Phil: And one question I would have is, if this gets to [00:24:00] what the intention is versus actual usage, is the usage of is the rubric from your perspective being used as a design guide or as an opportunity to evaluate course designs after the fact? And specifically the actual usage today in the transition from emergency room teaching to online teaching?
Jesse: I’ve seen the rubric over many years. I’ve seen it used in all manner of ways. I mean, essentially I’ve seen it used as an after the fact evaluation of effective online teaching, as though we can do that neat and tidily. And I think that that’s one of the issues that I take. And I’ve also seen it used as a starting point, a place to inspire new or existing teachers to think about their online learning in different ways.
I think that the the problem comes when I see a rubric like this being weaponized [00:25:00] by administrations. And I don’t use that term lightly. I mean, I use it very thoughtfully because students are feeling precarious right now, but teachers and faculty members are also feeling precarious. And when a tool like this is used as a quality assurance mechanism, it ends up feeling like it is an instrument of an administration looking to control teachers.
That’s at least the way that teachers often feel about it.
Phil: I guess part of the thing that’s interesting to me is so we’re so much talking about the usage by school administrators and particular at this point in time, even though the issues have gone beyond this point of time. But does the rubric self encourage this [00:26:00] usage, or is this something, the overusage, the weaponization, is that something that’s completely separate from the rubric – it doesn’t need to be this way? How does one lead to the other?
Jesse: That that leads me to a question that I’m often having with folks. The idea of is a tool neutral to its use. And I think absolutely, yes. I think tools have pedagogies baked into them. Tools teach us how to use them. And so I think rubrics in general – I mean, we could have a whole other conversation about rubrics in general – but if I think about this rubric in particular, the way and this is a lot of what my blog post in my keynote were about, and not just targeting Quality Matters, but targeting a way of thinking about education.
And ultimately, I think the issue is that when you take a 42 or 43 point rubric and that what it is, is when someone looks at this, even when they look at it visually without actually reading the words on the page, it feels immediately inscrutable. It [00:27:00] feels like the mechanism or the instrument has some sort of inherent wisdom that the person looking at it couldn’t possibly yet grasp. And when they look at it, they feel overwhelmed by it. A lot of people talk about the goal of rubrics being to simplify or to make transparent what is otherwise tacit or unspoken. But I think that this one does something different. I think when you see it, you feel overwhelmed. I was going to say you feel terror. Honestly, I think that there are some faculty who feel terror when they see it. And so I think using that word is I’m not exaggerating. I know faculty who when they see this, they feel terror. They feel this isn’t something I could possibly grasp or have entrance into. And so if they look at the tool or instrument or mechanism, and they feel this is something I have no power over, this is something that only has power over me. There is no point of entry for [00:28:00] me into this conversation. It is just a large page filled with text, and no white space on the page for me to fill with my own thinking about what good online learning might be.
I think then what you have is you have a tool that is over architectured, and an instrument that is telegraphing its purpose and its function too overtly.
Phil: Now, I realize after asking you the question that was that was a softball question for you, because now you could go into the LMS. But are you aware, have you seen their Bridge? They have a Bridge tool as well. One of the things I’m hearing from Quality Matters is the rubric itself is not intended to be a tool to help faculty brand new to online teaching to be used, and [00:29:00] here’s a tool that can help you there. Have you seen any of the non-rubric tools that they offer?
Jesse: You know, I’ve seen quite a few of their tools. And over the years I’ve I mean, I’ve researched Quality Matters, and I’ve looked carefully at their marketing. I looked carefully at what they say they’re doing and what they’re actually doing. And I’ve approached this from the experience of someone who just – and it’s not just about their website, but I’m saying this somewhat metaphorically, someone who just lands upon their website, who lands upon their work, and then where do you travel from there? Ultimately, the point of entry for most people with Quality Matters is the rubric.
So the idea that there’s something else beyond the rubric is it just isn’t how it works. In practice, people approach Quality Matters through the rubric. That’s their entry point. And so that’s the first page of the book. That’s the first page of the story that Quality Matters is telling. And one of the things that I think that happens with the way that Quality [00:30:00] Matters their entire oeuvre of materials, which is there’s just a wonderful wealth of stuff that they have. I think that the folks at Quality Matters are incredibly well intentioned, and they’re doing really good work to think about what online learning is and how it works. But I think what happens is you come at it through the rubric, you come at it from this place of fear, terror, feeling overwhelmed, feeling afraid that you don’t know what you’re doing.
And then there’s a way in which the Quality Matters ecosystem is going to be an answer for you. And they draw you deeper and deeper into this ecosystem. And there’s lots of stuff there, lots of useful, productive stuff. But it isn’t how I see Quality Matters being delivered, if you will, on the ground. Any time I’ve ever been to a workshop that is employing Quality Matters, the rubric is what is led with. And I think that that’s a story that Quality Matters [00:31:00] can rewrite as an organization. But I don’t feel like they’re there yet.
Phil: Now what we’re saying is it’s very much the how things are implemented in reality in actual workshops on campuses, and also to narrow down the conversation to what you said, it’s in particular for teachers or faculty who have not taught online in the year 2020. So within this context, one of the questions that comes up is if this rubric, and if the usage of Quality Matters is not working because it’s overwhelming for the reasons you’ve laid out. What are better approaches to help these new faculty or faculty who are new to the modality and thinking through these subjects, including accessibility? How do you ensure that courses are accessible? What is a different approach that is much more realistic [00:32:00] to what these faculty needs are?
Jesse: I think and I’ve said this in lots of different ways, in lots of different forums, but I think we need to start with hard conversations about what the purpose of education is, who our students are, who we are as teachers at our institution, who are our faculty, what strengths, what wisdom’s do they bring to the work? And we start from a place of figuring out not what are the two best practices that are going to work for everyone at every institution. But we start by having the hard conversation about what specific approaches do we need to employ at our institutions in our context, even in our disciplines, even in our specific classrooms.
And I think that that’s those conversations get frustrated when we employ approaches that rely on best practices, because really, when I think about best practices, what I want us to talk about is good sometimes in certain contexts with certain students and certain faculty member practices. [00:33:00] And we can only get to that place if we start with a conversation. So if I think about a rubric, I think that the best place to start with faculty, would be having faculty in a discipline, in a department, at an institution, writing their own rubric, having a blank space, an empty page to fill where they determine what’s important for their students at their institution.
Yeah, thank you so much.
Phil: Great. Well, listen, I appreciate your time on this, and certainly we’re going to refer to your blog post and keynote address for more rich description of some of these subjects.
But I appreciate your your time on this. Thank you.
