The Community That Teaches Languages and Powers Duolingo

[00:00:04] Announcer: You’re listening to Community Signal, the podcast for online community professionals. Sponsored by Vanilla, a one-stop-shop for online community and Localist, plan, promote, and measure events for your community. Tweet with @communitysignal as you listen. Here’s your host, Patrick O’Keefe.

[music]

[00:00:28] Patrick O’Keefe: Hello and welcome to Community Signal. This is being recorded on the Thursday afternoon following election Tuesday in America. CNN and The New York Times currently have Joe Biden at 253 electoral votes and Donald Trump at 214.

For two days, Biden has steadily reduced Trump’s leads in Pennsylvania and Georgia, while holding onto leads in Arizona and Nevada. If Biden secures Pennsylvania, or any two of Arizona, Georgia, Nevada, or North Carolina, he will win the presidency. It’s looking like that will happen, but you never know. We have to count every vote, and the Trump campaign is currently trying to impede that process, at least in the states where it might benefit them. By the time this episode is released, my hope is that the election has been called for Joe Biden.

On this episode, we’re talking with Kevin Reaves, who works in community for Duolingo, which helps people around the world learn new languages. They have a big ambassador super user program that we’re going to explore.

Our Patreon supporters are a group of listeners who back the show financially, which helps us to produce our program. If you’d like to become one, please visit communitysignal.com/innercircle for more information. Thank you to Jules Standen, Heather Champ, and Carol Benovic-Bradley for being among our backers.

Kevin Reaves has been working in community management for seven years and since late 2017, he’s been a contractor at Duolingo as a community support specialist. Previously, he spent time at Space Ape Games, Enthusiast Gaming, and The Huffington Post. Kevin, welcome to the show.

[00:01:50] Kevin Reaves: Thanks for having me.

[00:01:51] Patrick O’Keefe: It’s my pleasure. We’re recording this on Thursday after the election. How are you feeling? Are you doing okay?

[00:01:58] Kevin Reaves: I’m doing as okay as I can be. It’s kind of like the Groundhog day of elections.

[00:02:04] Patrick O’Keefe: Yes, I understand that. I know that you spent a little bit of time over at the Huffington Post. At times like these, I feel for our colleagues who are moderating comments sections, and political spaces right now. Whenever something happens politically related, I always think about those people, and I have several friends and acquaintances who work in the media and I’m currently doing some work with CNN. I’d imagine you were at HuffPo for a bit and you are happy not to be there right now.

[00:02:26] Kevin Reaves: Oh, yes I’m glad not to be in moderating political comments right now. It’s just that when you think about it. it’s a lot more divided now than it was then.

Actually back then I thought, “Well, wow, our nation is divided and you have to deal with a lot of abusive comments.” I can only imagine how things are now to be in those comments. We get a good idea when we look on Twitter for instance.

[00:02:50] Patrick O’Keefe: Yes, it’s– Trump is unique in many ways, all of them bad, in my opinion, but one of the things that he does is send these signals. He’s a community leader, whether we like it or not like, he is a community leader and he signals behavior for people to take cues from. All of these platforms now find themselves in this position of saying, okay, traditionally, not always, but traditionally you could look to leaders in the communities, people who would at least pretend to have good discourse or kind of good principles of discourse, even if they behind the scenes work to undermine you or were, I don’t know, even racists in private, they didn’t necessarily project that in public. They didn’t necessarily use the words and the terms that maybe they use in private where this guy doesn’t seem to have those constrictions or doesn’t seem to have those constraints.

He signals behavior and people take a lead from it. Now platforms have this choice of either apply our guidelines consistently, or create a double standard where you allow this person to behave in ways because they’re a political figure or an elected official, and we’re going to take a greater good approach to allow them to speak. You get more of that bad behavior though because the exception only applies to that one person or this small group of people. It’s like a bizarro society, because it’s like, you’re supposed to take an example from your leaders, but really the leaders should take an example from maybe the citizens in a lot of cases, or it’s like an upside-down behavioral influence thing and it’s frustrating for people who have built community.

[00:04:21] Kevin Reaves: Yes, it’s totally negative. I find myself all the time thinking, “This comment should be moderated” but Twitter has actually been doing a pretty good job lately, like for instance, when there was misleading information, of editing that out and reminding the users that, “Okay, this comment isn’t factual,” but it’s great that you mentioned the community and the community leader, the community leader needs to be mindful of things that they say in the discourse because it affects how the community acts. In this case, Trump has his own community of … Republicans that like his behavior and you find them growing more and more toxic. Their language is going more and more abrasive, actually adapting those misleading comments and behaviors.

[00:05:02] Patrick O’Keefe: Yes, I liked your pause there. Trump has his own community of … Republicans.

[00:05:08] Kevin Reaves: It seems warped from how Republicans used to be.

[00:05:11] Patrick O’Keefe: Yes, totally. I’m going to keep an eye on the election as we talk now but let’s get into Duolingo. I want to talk about especially the ambassador program, the Global Ambassador program. I want to learn about that, because from my own research and looking at the Duolingo site, from talking to you, it’s quite a large program. Ambassadors are vital to Duolingo, and to the existence of the service. I want to talk about that program and all of its different facets. First, what do ambassadors do for Duolingo?

[00:05:37] Kevin Reaves: We have plenty of ambassadors, they’re actually separated in several branches but they’re all volunteers that aid us in our mission to bring free language to the world. Whether we have our forum moderators that actively review forum content, and are really forum leaders because they have to interact in the forum and discuss things and actually put forth that measure of okay, how the ideal forum user is and it actually affects the community a lot.

Our course contributors are those who actively build our volunteer-based courses, we have a lot of courses on Duolingo, as you may have noticed when you looked at the Incubator, and a lot of those are community-driven. They have people from all walks of life and in their free time, take the time and build language courses for us. It’s amazing the amount that they contribute to our community. We also have educators, which are our glorious teachers, which actually teaching right now in the time of COVID has to be difficult, but they actively use that schools platform. In using that schools platform, they also really are ambassadors to others and their separate teacher communities, like within their schools of showing, “Hey, you can use this for your language classes,” or they go to conferences, and they bring about the good word of Duolingo to others. Lastly, we have our event hosts, our event hosts are absolutely amazing. That’s actually the branch I haven’t had much chance to work with. They actively host events, they normally would be events in person, but now they have to be virtual. They’ve shifted greatly towards that. They have little activities and things that people learn language. It’s a lot of interaction, that, hey, I’m gifted in this language, you can meet others in this community, they are learning a language just like you. We can all discuss the language itself and build from there.

[00:07:21] Patrick O’Keefe: Is there another group that’s new, called insiders. Is that something else as well, that’s coming along?

[00:07:26] Kevin Reaves: Yes, it’s coming along. Our insiders branch are mostly those users that are, I can say pillars of the community. If you notice someone is doing outstanding in the community, they have an amazing streak. Each day, if you have a certain amount of XP, that’s your head set up as a baseline, you will then receive one day towards your streak. Some people have 900 plus day streaks, 2,000 day streaks, those people are amazing. They really put forth the effort to learn. We try to reward that as much as we can. Those people are insiders with extra status, they may have extra access to say a beta program, things such as they receive extra information about the trial, upcoming features, or things.

[00:08:12] Patrick O’Keefe: When I hear that I hear five facets of things that drive adoption of Duolingo. The forum moderator is ensuring the quality of the conversations that exist around Duolingo, course contributors are creating the courses people take. I would guess, I don’t know, the reason that most people venture into Duolingo territory is because they are looking to learn a new language. It’s those courses that they’re taking, that are really the first lure in. Event hosts are bringing people together in a different way. Maybe they don’t want to be on the forum so much, maybe they’re more comfortable in an event environment. Educators or people who are driving adoption of Duolingo by bringing into classrooms and teaching environments. Then insiders are market research in a way for driving Duolingo forward in the future, as far as new features, new ideas, probably the most hardcore users of the product in a lot of ways from what you’re saying. There’s a lot of business benefit there. I want to talk about it more kind of in a couple of different segments.

First, how many ambassadors are there? I think you told me there’s over 300 forum moderators alone, how big is the program as a whole?

[00:09:09] Kevin Reaves: We are talking in the thousands.

[00:09:10] Patrick O’Keefe: Wow, what’s the biggest group out of those?

[00:09:12] Kevin Reaves: The biggest group out of those are event hosts, we have a lot of event hosts.

[00:09:16] Patrick O’Keefe: Why do you think that is?

[00:09:17] Kevin Reaves: Well, we did an amazing job of building the community of event hosts to be honest. It grew from a small number of events to an amazingly large number in a matter of years and it gets a lot of attention, which is great because you’ve always found a need, for instance me being a natural forum guy, I’m on the forums a lot. I always notice people say, “Well, how can I interact with other people and practice language or discuss language?” Really, Duolingo events is the one way that they can easily do that. People naturally cling to that.

[00:09:48] Patrick O’Keefe: What you’re saying is that people go to that product, they learn the language, but they don’t maybe know anyone that they can practice speaking it fluently with and have someone just kind of course-correcting that knows the language well. They’re eventually given the opportunity to talk to someone who knows that language, who’s fluent in it, and who can in a friendly way, say to them, “Hey, that’s actually this way, or that’s the masculine or that’s the feminine or that fits this way, or whatever it is,” into that particular language.

[00:10:13] Kevin Reaves: Yes, exactly. Have little activities and different things to make the whole process fun.

[00:10:18] Patrick O’Keefe: That’s really interesting because a lot of communities would be probably heavier on the forum moderator, content creator side of things, especially when you’re talking about an internet or a digital product and then have event host because event host feels like such– We feel like the work of going in-person pre-COVID and even hosting an event digitally during COVID and post-COVID is more of an involved thing that we have to do. We have to go set up, again, this is pre-COVID. Set up whatever. Set up the space. Set up the tables. Be there. Break through any awkwardness that people have as they’re standoffish as they come in the room.

It’s a different skill set. It’s a different thing. In this case, I was trying to think of other cases. Dating might be one pre-COVID, like dating events, speed dating. If you’re a dating app, bring people in-person, that’s a big deal. That’s how you’re going to probably for most relationships, really be able to take it to the next level is to meet in-person at some point. [laughs]

The same is true here though, because if you’re learning a language, you’re learning to better yourself, for sure. Maybe you’re only learning it for yourself, or for some niche need you have of understanding something. I would guess most people are learning it because they want to use it because they’re going to be speaking to people who have know that language.

[00:11:28] Kevin Reaves: You’re right, a lot of variety different reasons. I have people, “I want to visit this country one day. I would love to learn this language, at least get started in it.” We’ve even had people who, let’s say their grandparents came from knowing this language. They don’t know it themselves. They actually are an immigrant. We have warm stories about this. “I can now communicate with my grandma or my great-grandma using her native language,” even if it’s just a little bit that they learned. It’s a touching story and it really goes to show a lot of what language learning as a whole can do for people.

[00:12:00] Patrick O’Keefe: That’s amazing. I’d like to take a moment here to talk about our great sponsor, Localist.

Localist is an event marketing platform that aggregates, automates, and analyzes all of your virtual, in-person, and hybrid events so you can grow and engage your community. Their platform allows you to centralize your event calendar, automate email and social media promotion, and measure and analyze the ROI of your events. Localist integrates with your existing tools and you can even predict future event success using their Event Reach and Event Score features.

Find out more at localist.com/communitysignal and take your event strategy virtual with Localist.

For the event host, does Duolingo provide them with a in-suite in-product tool to host events or they kind of lean towards something else, some other tool like Zoom? How are the events being powered right now during COVID?

[00:12:45] Kevin Reaves: Currently, they are being powered through Zoom.

[00:12:47] Patrick O’Keefe: Got you.

[00:12:47] Kevin Reaves: -for the most part. With Zoom is the last state we’re at right now.

[00:12:51] Patrick O’Keefe: Zoom just sucked up a lot of market share. I don’t know if it’s fully explainable, but there definitely was a right time right circumstance type of situation.

[00:13:00] Kevin Reaves: Yes. It came in perfect. I worked remote so I was used to using Zoom. Zoom is amazing. I actually want to use Zoom a lot more with other branches, for instance. I’ve been meaning to get into more Zoom calls for teachers. I used to sometimes have what I call a water cooler, talk with it every now and again to see what’s on your map. How’s the state of things right now?

[00:13:22] Patrick O’Keefe: How do you become an ambassador? How strict is that process?

[00:13:26] Kevin Reaves: Each branch has its own separate application process, actually, but for forum moderators and you and I can discuss this a whole lot, with our history in forums. We want a moderator to be at least 18 years old, for starters, because of certain content online. If they are a helpful user on the forum that does not have a history of issues such as warnings, et cetera and generally, if they’re active. I’ll say active for like a year, then we look in and we say, “Well, hey, this person might be a good fit.” They are vetted only by community staff. They’re also vetted by our form moderators themselves that are seasoned.

Our course contributors, of course, they have to be fluent in the language. We’ve actually been improving the application process for that so that we know, “Okay, what’s your history here with language? Exactly, have you studied language itself? Are you a native speaker?” We have ways of noticing that because they have to write things in the target language and in the native language. Filtering through those applications are sometimes it’s fun because you always sometimes we run into spam applications. Then you could just easily vet those. One thing I love about that is you see how passionate people are about teaching. I’ve encountered several applications where people are, “I don’t feel this language is taught enough and I would like to assist in any way that I can.”

Even people who are like, “I want to teach my husband this language, and I also have it on the side as an easy resource.” I think, “Well, wow, that’s amazing.” That’s a whole another reason. We look for experience in the language at least. We want to make sure that they have their hands on the pulse of knowing the language itself so that we can build the best courses possible. For teachers, teachers, their passion comes just straight forward. We always ask them, “Where do you teach? What languages do you teach?” It’s amazing how many multilingual teachers that we have.

We have teacher ambassadors that speak at least three, four languages, and me, a person who really isn’t a polyglot, I’m like, “Wow, that’s absolutely amazing.” Then they just display how they use Duolingo in their classroom. Like for instance, a person will mention that, “I’ve used Duolingo in my classroom for years. I’d like to know more. I’d like to become part of the educator network as we call it, interact with other teachers.”

[00:15:46] Patrick O’Keefe: Is the educator branch about giving them more tools and listening to feedback or they’re using it already and they’re saying, “I want to be a part of this thing.” What’s the benefit of that thing?

[00:15:57] Kevin Reaves: It gives a great communication with staff. One thing that is great is that it gives them an opportunity to interact with other teachers, the teacher community, and share resources, Duo resources. We’ve had teachers create resources for other teachers, such as, “Hey, you might want to use this flashcard system in the classroom.” It goes far beyond even our Duolingo for schools platform. They actually begin thinking of their own games and different things that can be used in the classroom, share it with other teachers, make videos and share it with other teachers, all to help benefit the way that we reach kids.

[00:16:32] Patrick O’Keefe: For a lot of them, it’s really just a very good way of connecting with other language teachers. You could assume that I could Google and find communities of language teachers that are out there. If they’re using Duolingo in the classroom and they find benefit from it, then there is almost sort of a pre-qualification to join the educators’ branch. They’re tapping into this network that they will hopefully come to trust for advice and information and knowledge about language learning and language teaching.

[00:17:02] Kevin Reaves: Right.

[00:17:03] Patrick O’Keefe: Then just to back up a little bit, so that’s educator benefits. What do you see as the benefits for the other branches? Why did they do it?

[00:17:11] Kevin Reaves: A lot of it for the most part is passion. Pure passion for what they do, but I can imagine how it is to possibly work on the side of creating courses and then to see your course, the lab and to actually see this many users are learning on my course. The stats are actually there when someone clicks on a language that they wish to learn from their course correction. Like for instance, 150K learners are in this course and they actually can see that and the benefit that it has. With forum moderators, one thing I’ve learned about them is, as I say there are people like me, passion for our forum community, reaching out with others, making sure that the forums are safe, creating new resources, they help people learn the forums or language resources for the forums themselves.

[00:17:57] Patrick O’Keefe: This shouldn’t sound cynical because it’s not. Are there any people who have somehow used Duolingo as a springboard to have a full-time something business? Like for example, “I created this course on Duolingo. It helped a million people learn this language. I am a language instructor. If you’re looking for someone to teach you the language in a more like one-to-one way,” is that something that platform allows for? Is that self-promotion? Do you see any people out there who are using it for a business benefit in their own lives?

[00:18:30] Kevin Reaves: I’ve encountered people that, for instance, we’ve had people write recommendation letters for graduate programs, for instance, in language and different things. That’s very helpful to have our word there that, “Hey, they helped us build this. Of course, they were actually the course leader in this course that helped that direct the efforts for that course.” It goes a long way for them so yes, I can definitely say that.

[00:18:54] Patrick O’Keefe: Let’s pause to talk about our generous sponsor, Vanilla.

Vanilla provides a one-stop-shop solution that gives community leaders all the tools they need to create a thriving community. Engagement tools like ideation and gamification promote vibrant discussion and powerful moderation tools allow admins to stay on top of conversations and keep things on track. All of these features are available out of the box, and come with best-in-class technical and community support from Vanilla’s Success Team. Vanilla is trusted by King, Acer, Qualtrics, and many more leading brands. Visit vanillaforums.com.

Is language learning like any other passion? For example, you’re a sports fan, you are a Lakers fan, you’re a gamer like you are and I am, a couple hours a day with Dead by Daylight and Marvel’s Avengers right now.

[laughs] Is it like that? I know one language, I’m one of those people who say, I wish I knew more, and yet I don’t try. I can’t, I have no one to blame but myself. My great grandmother could speak like eight languages.

[00:19:52] Kevin Reaves: That’s amazing.

[00:19:53] Patrick O’Keefe: She was just very well-educated. She spoke eight languages. She fled Europe during Hitler. She was Jewish. She was just an amazing woman and she did all these things. An amazing person and I got to meet her. I think she passed when I was eight. Go back to my question, is that community a lot like what you see in gaming, where you have people who love gaming, and they’re passionate about gaming, so they join IGN like you did, and they become active. Then maybe they become a moderator. Are there people who are equally as passionate about learning new languages and about broadening their ability to communicate with a wider group of people?

[00:20:27] Kevin Reaves: That’s definitely what I see on the forums every day. The community helps each other a whole lot. If someone has a question about language, or even whether or not they’re phrasing something correctly, or anything such as that, you can instantly see community members jumping in there and say, give them tips, even when it comes to studying or different things such as that, they are very passionate about it. It amazes me because many of them are learning several languages at once. For a guy like me, who is mostly going to say, “Okay, I’m going to practice one language at a time.” It’s just absolutely amazing seeing the effort that they put into it. It’s those community members that one thing I would like to do is, I’d like to focus on that a bit more. Perhaps see what we can do to mobilize them a bit more with what they already do, as far as let’s say, for instance, that we can have them as a forum moderator that specifically would interact and create resources in certain areas, such as a sentence discussion thread, where people have difficulty. It definitely is the same as say a gamer or any other community that’s passionate about, let’s say I like cooking. I want to dive into the cooking community and help others learn how to cook a meal or something like that.

[00:21:41] Patrick O’Keefe: Call of Duty and learning the languages of Europe, those are two sides of the same passion coin, perhaps, depending on the person you’re talking to.

[00:21:48] Kevin Reaves: Definitely.

[00:21:49] Patrick O’Keefe: Can you talk about the metrics that exist for measuring those segments of the ambassador program?

[00:21:56] Kevin Reaves: Oh, great. First, I can talk moderator metrics because that helps me a ton. For instance, for such a large group of moderators, you have to make sure that the moderators are active or check when they’re inactive, so you can know when to check in on them, or see if the forums are properly covered. Quarterly, the metrics are checked and we can see, “Okay. Who’s active? Who’s been inactive? Which forums have lost moderators, and which forums have gained moderators?” And see the number of moderators per forum.” For instance, let’s say we don’t want a whole language forum to have only one moderator, for instance, then that’s when we say, “Okay. This forum needs more moderators.” From there, we just bring into action and add more.

In terms of contributor courses, the main thing is that we want to make sure that we always have this goal of at least 90% of our courses are active at any given time.

If we notice that, a course is being inactive and we regularly run the metrics to see if one is inactive. That’s when we check in with the team and say, “Are there any blockers? Are there any way that we can help? Is there something that we can do to help in the building of this course?” We interact with them closely. They do a great job of bringing to us any blockers, any issues, any concerns that they have, or even improvements that they would like to see considered and we follow along with them.

[00:23:21] Patrick O’Keefe: You mentioned the 90%. When 90% of course is active. Are there simply languages that are unpopular, [laughs], or languages that people just, for whatever reason, a small percentage of people that speak that language, or it’s that you just accept that they are not going to be in that 90%? Are there any languages like that?

[00:23:39] Kevin Reaves: Yes. There are some obscure languages. There are a few because when you think about it, you have to think about anyone with any given base language and what language they’d like to learn. For instance, we have languages like Swedish for Russian speakers or French for Turkish speakers. Sometimes it’s hard to find contributors in certain ones because you sometimes can’t find many people who happen to speak both of those languages and are passionate about teaching both of the languages.

Our contributors do a great job, like our contributors to a great job of reviewing applications themselves and adding people. It’s when they can’t find anyone, or when a course becomes inactive that we step in, and we look and we decide to reactivate that course. We do things such as forum blasts, or different ways to go see well, “Hey, would you like to help out on this course?” Which a lot of times the forum users are very eager, because they will say if a course is alive, they will wonder, “I want this course. When is this course going to be released?” I know that or I can help out with this course and that gives them a great opportunity to dive in there and take part in that. There are some, for instance, that we understand if it’s hard to find a contributor for that time period you to reactivate the course. It’s something that we really review as time goes on.

[00:24:54] Patrick O’Keefe: Do courses need to be updated over time outside of errors or sentences that are reported or mistranslations, things that slipped through? Are there other reasons that courses are updated or improved or adjusted over time?

[00:25:06] Kevin Reaves: Oh, yes. One thing I like to see is that a lot of our contributors, they initially begin working on a new tree. A tree is like the complete set of courses with all the different lessons on it. If they find a way they can improve the course, they’ll start working on a new tree for the course, it isn’t visible. Once they start doing this, and once it’s all set up, then it can be tested with a group of users. From there, we can see and gather the retention rate of the tests and different things such as that. What feedback the learners give us, the alpha testers. From there we can see whether or not the course is published into an updated state.

[00:25:45] Patrick O’Keefe: Do you have a special group of people that you test courses with or is it more of a random selection?

[00:25:49] Kevin Reaves: We do things in two ways. Early on before a tree is sent into beta, we have our alpha testers, which assist our course contributors greatly. From there, we beta test in A/B test form. Some people will have the new tree. Some people will have the classic tree. Those with a new tree can definitely give us feedback on. The same thing goes for courses that haven’t been released 100% or in other words aren’t in phase three. Phase two is our beta phase. What we’d look at is, “Okay, did people remain in this course? Did they find the course too difficult and drop off after a certain lesson, for instance?” Or are people receiving a ton of errors and it’s one section of this course? Then those things are looked into.

[00:26:34] Patrick O’Keefe: I was curious about how you ensure the quality of language courses in a UGC environment. You just answered some of it, I would say, which is that you connect people who are qualified or believed to be qualified for those languages to a need, you have Turkish speakers learning French, for example and then they started. I imagine a course can start as basic as a simple course and then you have testers that share, that go through it, provide feedback. Then you expose it to the larger community of Duolingo users and they give feedback. Then it gets improved and the test can be made more advanced over time. Is there anything else that is part of that quality process that you’d like to share that ensures that ensures when a course reaches the final, I don’t know if there’s a 1.0 or what you call it, but past beta, when it reaches that stage that you feel really good about that course?

[00:27:23] Kevin Reaves: Basically, if we notice that people are able to go through the course, and we have a high retention rate, then it’s basically safe to publish. Also, we try to keep it uniform throughout. Let’s say you want to teach the basics. We have, for instance, little lessons such as animals. You want to learn about animals everything We keep the subject simple, and it gets harder as you go along through the checkpoints. Because every course is built in this similar fashion, we have a uniform approach to our courses. So that one isn’t necessarily completely different, like one doesn’t start with, “Hey, you’re going to start talking about schooling instead of learning about verbs, for instance, first,” and it goes a long way.

We also use a common framework. One thing I love about our contributors is that they are very good about following the norms that we have in place and we have many mentor contributors that have been around, very seasoned. If there’s a new course leader, or new courses being built, they happily dive in there and help them with the tips they use in growing their own courses, and help guide them along to ensure that theirs is of quality as well.

[00:28:34] Patrick O’Keefe: Contributors have their own community where they can help newer contributors or they can ask questions to more experienced ones?

[00:28:40] Kevin Reaves: Yes. Well, that’s what often happens in our Slack group. They have direct contact with our community staff members, and also their fellow contributors.

[00:28:49] Patrick O’Keefe: With a course, you mentioned retention rate, that’s if they finish. Is that what it is, if they finish the course?

[00:28:53] Kevin Reaves: Yes, basically. Because if they drop off, then it’s like, “Well, this person didn’t make it past this point in the course itself.” We’re like, “Well, we’re not retaining a user in this course, a learner in this course, for instance.” We have to take that into account of the course quality and also the difficulty of the course because we want to make sure that is step by step learning, and not too difficult for them.

[00:29:18] Patrick O’Keefe: In a program so large, you mentioned thousands, how do you monitor ambassadors to ensure they’re being good stewards within the community, to ensure they’re being active? You mentioned with four moderators, you have quarterly metrics review. What other processes have you baked in to help manage such a large program of thousands of people?

[00:29:37] Kevin Reaves: For one, it’s a lot of people power for instance. We have a small community team. For instance, we have several people who work with different areas. For instance, myself, with moderators, I work with one of the senior community managers and a fellow community manager with four moderation, we regularly meet up and discuss, “Okay, what’s new here?” Or review everything that we need to their interactions, for instance, to ensure that they are properly supported. Their questions are answered. Their feedback recorded and they’re both addressed.

They do a great job of pinging us whenever they run into something so that we can instantly notice. I will say that a large part of my day beyond my other functions is looking into our Slack groups and seeing what’s going on because, while we can be on the forums, I love telling this to learners on the forums, I’d say we have an active presence on the forums. Our moderators do a great job of also bringing attention to things that we may not notice at any given time, given the large community. The same goes with the other branches as well. I have teachers that often bring to me books that other teachers have, that they might have encountered on the forums.

We have an educators forum as well. We have a few helpful teachers that can say, “Hey, Kevin, we noticed that this person’s assignment isn’t being recorded properly. Is it possibly user error or is it a bug?”  Then I’m able to instantly look into that and make sure to disperse properly. It’s basically, you have a large set of community members all helping each other out, bringing things up to community staff, and community staff’s placed in strategic areas. For instance, I contributed because they are a large base. We have at least four community people on that alone. It really helps out a lot. Of course, as with any community, we’re constantly streamlining processes.

We’re constantly trying to evolve as that is important, whether or not it deals with tooling, extra tooling that we may have to grab those metrics and things, so it’s before we manually notice something or better reporting tools.

[00:33:56] Patrick O’Keefe: When I hear that I hear recruit well into these groups. That you can trust the people that are in the programs to point things out to you, pay attention to the signals that are happening in the Slack groups, in the forums. Often times the loudest signals, what’s good and bad, that are happening in the community and as best as possible, have routine metrics and tools in there to regularly report data to you that will signal additional red flags or issues that you need to look into. Which makes a lot of sense.

You mentioned tooling. You told me before the show that “community tooling is a challenge that has been on your mind. As community professionals, we often dream. I know I do, of tooling that would make aspects of our job easier, whether it’s in terms of moderation, tracking statistics or automation.”

These are your words still. “The difficulty is showing the impact that these tools may have to increase productivity and dramatically improve community efforts.” What are you missing right now that would make your life easier?

[00:32:41] Kevin Reaves: I would say a lot of admin tooling, for instance. Forum moderating tools, they make things simpler. With, especially during this time period, a lot of people are indoors. There are more people on online forums. There are more people sending comments and messages or, this includes those abusive messages and we want to make sure that for instance, spam is cleaned up as fast as possible. That accounts that attempt to come back over and over again, that we have proper ways of tracking those accounts and being able to notice, well, here, they’re using the same email type, for instance, or the same IP address.

One major thing that I think would be a huge help is, let’s say that we want to track what our moderators do for moderator oversight. Easily able to see when this person has been warned. Who they’ve warned by. The warning message, for instance, information like that. Who’s our most active moderators? Things such as that so that then we can not only better moderate the forums, for instance. We can also reward those who go above and beyond for the call of the community itself.

[00:33:47] Patrick O’Keefe: One thing that you mentioned that I think is really great that and people who make software listen to this show, sponsor the show, et cetera. That’s why I like to ask questions like this, what I’ve called when I’ve talked to people who build software, forensic science. I want to bring forensic science to community. You mentioned being able to know right away without having to look, when someone uses the same IP, when someone uses the same email address. When someone has similar profile data, for example, without you having to search and reconcile it yourself or go into the database and run a query.

If you know how to do that and if you have access to the database, which most people do not have access to the database and just the software would automatically look out there and not necessarily take action but just flag it for you. Put it in a dashboard and say, “These three users have the same IP.” Then you can decide what to do with those people. Maybe it’s incidental. Maybe they’re all in the same classroom. Maybe they’re all in the same family. Man, you’re like, “Okay, this is fine. There’s nothing bad. They’re not bumping each other up or saying, “Oh boy, you’re smart.” “No, you’re smart.” Boosting each other up. Tools like that, as far as I know don’t exist. I haven’t seen them. They might exist but I haven’t seen them in the easily available community software.

[00:34:57] Kevin Reaves: You’re right.

[00:34:59] Patrick O’Keefe: I would love to see them. I think pattern recognition and reporting that pattern to you would be super helpful.

[00:35:05] Kevin Reaves: It works wonders and especially now because like I said, people have a lot more time on their hands because they are avoiding public places. That gives a lot more time for community mischief, as we probably see in other communities that we just actively use on our own. For instance, someone might be a chronic spammer and it’s like, I want to get rid of this spammer as soon as possible or possibly automate a way for this spam to be detected and automatically removed or automatically put into a queue for you to simply press and remove. That works wonders and it also helps the moderation team because that’s less manual moderation on their part.

[00:35:43] Patrick O’Keefe: I probably sound like a broken record to people who have been listening to this show for a long time. I’m still surprised we haven’t seen more innovation around the word censor feature on community software.

[00:35:52] Kevin Reaves: That’s true.

[00:35:52] Patrick O’Keefe: 15 years ago I had someone write a feature for my communities where if it hits the censor, the post is halted and the person is given a chance to edit their post with the issues highlighted in it, so that they could adjust it and repost. That doesn’t work for all communities. Not everyone should have that turned on. Some would be that would be a trigger for abuse. That would lead to other issues but in my communities, they’re at the right scale and environment where that stopped removal for profanity. Almost 95%, 98% we don’t remove stuff, profanity anymore.

People update their posts themselves and they submit the post and generally, people like it because they can do it themselves. They keep their posts up. We never have to remove it. They never have to have their posts removed. They never have to hear from a member of staff and we don’t have to do that whole process of documentation. It skips the whole thing. There’s no violation of the guidelines because the post is never made and your mind would say, “People will circumvent that,” and yes, in some communities, that would be a major issues. In many communities, no. People will adjust their posts and hit submit.

[00:36:47] Kevin Reaves: You’re right.

[00:36:48] Patrick O’Keefe: Yet the word censor feature, as far as I know, remains practically unchanged in every form software and community software that exists for the last 20 years.

[00:36:58] Kevin Reaves: That’s one of the main things that we’d like to have, especially for, let’s say for instance, profile names. So often, you can run into an inappropriate username and it’s like, “Okay, well, this person created this username and whether they’re aware of the guideline or not. If they were given the instance upon account creation to know, this name includes profanity, or something else that isn’t allowed. Let’s say it’s something like, a name, Adolf Hitler or something so there’s this– Something that you definitely don’t want on the forums. Then they have the opportunity to corrent that immediately before they even create their account and then, we can just go from there. It should cut down a ton of issues.

[00:37:35] Patrick O’Keefe: Yes. Even the censor being applied to the username. That’s something you see, unless you join PlayStation Network, for example. They have a term list of things you can’t have in your username. That hits on the side where most impact is made, which is, let’s say 99% of online platforms are people who do not have a thousand dollars to spend on software. Not even a thousand dollars. They can maybe afford to pay couple hundred once upon a time for vBulletin. These days for maybe an Invision, or XenForo, or they can afford to host a Discourse, something like that. I remember when I was running phpBB 2, and I had someone write a hack so that our filter would apply to the usernames.

[00:38:16] Kevin Reaves: Oh, that’s crazy. [crosstalk].

[00:38:17] Patrick O’Keefe: Because that wouldn’t exist on and so, things like that. Administrative features are some of the most just underserved features that exist in community software.

[00:38:28] Kevin Reaves: You’re right.

[00:38:29] Patrick O’Keefe: I’m just waiting for the call for someone to hire me, or you and be like, “Hey, we need the community person on this side and just tell us what to make, and let’s make it.” Because, I know, I’m not saying, Discourse has great community people. Sarah Hawk‘s great. They’re out there, but there are features that are out there but you mentioned COVID a couple times and on our most recent episode before this one with Paul Bradley, we talked about the idea that perhaps people might be more patient after COVID-19, when it came to issues in online communities. That was not a proven theory. It was just sort of an idea that we were discussing and you told me before the show, and you even referenced here that you’ve witnessed, “An increased amount of abuse after places underwent lockdown measures, or people decided to avoid being active in public places, not only when it comes to actual abuse cases that you log, but also concerning posts such as potential self-harm.” What are you seeing?

[00:39:15] Kevin Reaves: There’s a lot of this. There’s an increase in spam, or for instance, let’s say as more schools use Duolingo naturally because of virtual schooling, those students enter the forum, we often have teenagers, younger kids, they spam the forms with guideline-breaking type of content, and there’s a lot of that so we see a lot more spam. A lot more nonsensical entries. Things that need to be deleted. We see a lot more posts for instance, that are attacking others, and as I mentioned before, self-harm posts. I hate to use the word depressive but that’s what it is. It’s a very impactful period emotionally for people. We mentioned earlier the whole climate with elections. We have a very divided nation, for instance, here in the US.

The world itself has been a bit more, I’ve got to say divided, in many ways and then now, on top of that, people aren’t able to do many of the things that they enjoy. Let’s say you are used to going swimming with friends, or playing basketball with friends, or going out to a movie and things such as that. We often see these posts where, not all of them are immediately self-harm. A lot of them are just enough to raise a flag to where we feel that we should automatically reach out. We not only send emails to these people to check in on them, but we also often send a message inside that thread before we lock the thread, or delete the thread so that they would get notification in the email inbox, let them know that these are the hotline numbers and, we’re looking out for you and the importance of self-care as a whole.

We can also bring that back to volunteer teams and to ourselves as community professionals that see these things daily. Because, often, a lot of people don’t realize is that, the impact of seeing these things each day, seeing abuse, it can wear on the body a bit. It can wear on you emotionally and that’s why it’s important to sometimes take that break, get yourself a drink of water, walk around a bit, maybe take a walk, or something. If you can, listen to some calming music, perhaps read a book for 30 minutes and then you start back on the work that you’re doing because, the major thing is that, it is great for a community such as this. There’s a sense of understanding because, everyone’s going through something similar because of the pandemic situation. It’s easy to find support within volunteer communities, and it’s easy to find support within community teams, with teammates that are understanding and things. What we do is it’s good to highlight the positives and come together and discuss those, lean on each other, and give each other advice, because it goes a long way when it comes to online community, especially with professionals.

[00:42:06] Patrick O’Keefe: You make a really interesting point and one that I hadn’t really thought of coming into this is just the volume of students now using Duolingo, because of COVID. What that audience means, what stresses they’re facing, how they might deal with those stresses, the vulnerable groups, they might exist within being a teenager, just because they’re on Duolingo, how that might filter into the community and how you need to not only take care of it, but also what your responsibility is in doing so. I have to assume a lot of communities that are student-focused, not the Duolingo student-focused, but that receive a lot of students, especially elementary and high school students, and college students are probably seeing similar things right now. That’s good to be aware of and for you to point out.

[00:42:49] Kevin Reaves: What it is it also allows us to expand our protocol, because of how things are during this time period. What can we fine-tune to ensure that these messages are addressed correctly, or for instance, I think of things such as if it’s a student, if I know this person is inside a classroom and I can see where the teacher account is. Reaching out to the teachers themselves, letting the schools know so that the parents also are aware. Things like that, it goes a long way.

[00:43:22] Patrick O’Keefe: Kevin, it’s been a pleasure to have you on. I’ve appreciated the conversation. Thanks for spending some time with us.

[00:43:27] Kevin Reaves: Anytime. I’ve really enjoyed it.

[00:43:30] Patrick O’Keefe: We’ve been talking with Kevin Reaves, a community support specialist at Duolingo, that’s duolingo.com and connect with Kevin on LinkedIn, at linkedin.com/in/kevin-reaves-24b25657.

For the transcript from this episode, plus highlights and links that we mentioned, please visit communitysignal.com. Community Signal is produced by Karn Broad and Carol Benovic-Bradley is our editorial lead. Take care of each other.

[music]