(In case anyone is wondering why this is posted in the Panopticon rather than Announcements, it's because the forum permissions in Announcements don't allow replies - it's not very good form to solicit replies in a forum where nobody can reply!)
I think every admin (and former admin) would agree that the part of the admin job we all like the least is the part where we have to consider discipline (frankly, the only parts of the admin job that are even slightly "fun" are the opportunities we get to announce upgrades, new features, etc. to the community). We do it because we love the community - if there's no discipline at all, the game and community will eventually be overrun by those who don't want to play by the rules (such is, sadly, the nature of the internet).
After seeing how things went wrong here, we have done some self-examination and also sought feedback from some non-admin sources to help us figure out how the process can be improved. What I'd like to do is walk everyone through the process that we had been using that got us to this point, ask for feedback from the community to make sure we're on the right track in improvements, and hopefully make some lemonade out of this lemon of a situation.
Before we begin here, I think it's important to note that we feel it would be very unwise to "publish full logs" or describe exactly how our investigative tools pick up what they do since the moment those get out, anyone who doesn't want to play by the rules will have a pretty good idea how to avoid detection - we hope the community understands the need for this particular bit of opacity. That means you won't find detailed logs in this thread, and you're not going to find me trying to make a point-by-point comparison with pyroshroom's thread because I think that lends itself to an adversarial "no, you're wrong because X" type of post and I don't think that's going to be helpful right now; instead you're going to find me talking about the patterns that arose out of analyzing those logs and giving a general look as to how the process unfolded from the admin side of the table.
I'm going to lay things out "as they happened" in kind of a stream-of-consciousness style without commenting inline about where process improvements could be implemented; process improvements we feel could be implemented will be noted in their own section, so if you see something and think, "hey, that's not the best process" and you don't see it in the "So... What's going to change?" section, I welcome you to point it out.
I also feel I should thank Southall for his gracious permission to share the below; a version similar to what you see below was shared privately with him to make sure he was okay with this information being published since a lot of this centers around him.
The General Process for Admin Disciplinary Investigations
Usually things start with a report from a player that, "hey, X looks suspicious, can you please look into this?" One of the admins (usually the one to whom the report comes) checks through our log tools and attempts to determine whether or not the complaint is a legitimate one and surface any information that might condemn or exonerate. Once the admin has completed the check, he posts the relevant logs in an admin-only forum dedicated to reviewing incidents and asks other admins to check his work for errors before proceeding.
Sometimes, the investigation finds there is nothing going on at all. These cases are usually closed quickly.
In some cases, infractions are obvious and indisputable ("Character A raided Faction X and then Character B belonging to the same player raided Faction X twelve hours later" or "Character A has been making unwanted, sexually explicit comments to Character B for a week after Character B told them to knock it off"). In this case, sign-off on a suspension is generally a formality, though there may be discussion on the appropriate length. There are not usually arguments about this stuff - the violation is crystal clear.
In some cases infractions are NOT obvious, because they may have taken place outside the realm of what we log (for example, we don't have a method to log a faction's discord channel), but we may see in our logs evidence I would describe as "unusual behavior patterns" that would imply something suspicious is afoot. These are of course the toughest cases to adjudicate because we have to use our best judgment to come to a conclusion based on evidence we know is incomplete, but these are ALSO the way in which the most serious infractions usually occur.
The Recent Oblivion Squadron Incident
A report came in on 7/16 that Oblivion Squadron's reactive ability in raid defense seemed suspect citing one character that always seemed active in raid defense across a wide range of hours. An administrative review of raid logs prior to 7/16 was conducted by plscks, who did not find any obvious evidence of activity that seemed out of the ordinary, but it did mean that attention would be paid to the next couple of raids just to be sure we had a "nothing here" finding.
When the logs of the next couple of raids did come, the following observations were made:
Logs of Southall's character, a veterinary assistant, showed that Southall was connected and very attentive starting about 10 minutes before the IB raid on OS that occurred on 7/22 around 1600 Game Time.
Logs also showed that Southall was active on 7/27 ten minutes before an IB raid at about 2000 Game Time and quaffed a Potion of Invisibility before standing outside to watch.
Procedure was to make a decision at this time based on the log evidence so as to not tip players off that an investigation is underway... I'll ask you to put yourself in the admins' shoes for a moment here and ask "given only this evidence, what conclusion might you reasonably draw?"
The admins discussed the evidence we had and it was agreed that the first data point alone was NOT necessarily suspicious, it could have been coincidence; however, adding in the second data point made the behavior much more suspicious. It seemed unlikely that the same character would just happen to be active ten minutes prior to consecutive raids by the same faction, spaced five days and four hours apart on the game clock, and that the second time the player even had the presence of mind to take an invisibility potion and step outside to watch the raid party arrive. We tried throwing some possible benign explanations around but none felt convincing - the only thing we could come up with was, "he MUST have some insider information."
(At this point, I am going to make a digression to add some context to this discussion. As most raiding characters know, one of the most thrilling things you can be a part of is an active defense of your faction during a raid. It is also one of the things that is most frustrating for a raider to encounter. As some know, we have had issues in the past with players abusing "insider knowledge" to make sure they actively defend their favorite factions and it became so frustrating to the player base that we lost players because we didn't take action against the cheating player quickly enough.
Because of this, we're very sensitive to "insider info" and one of the things we tried to adjust in the transition from Breath 4 to Breath 5 was limiting the information available to raiders and raidees outside the message pane of your individual character - we removed the "There are X players currently active" counter since players would watch for spikes in that counter to anticipate an incoming raid. We also changed the way characters are displayed - you'll note that when you are dead your character page no longer says "This character is a formless spirit, floating above the planes, dead." We did this because we knew some raiders would look at potentially defending petmasters to see which factions had dead petmasters and thus would be easy prey - this usually meant, "factions that had just gotten their stronghold back up" and it's no fun when someone deliberately kicks you when you're down.
When we migrated to PHPBB, we didn't think about the 'Who is Online' because, well, PHPNuke works differently than PHPBB does - PHPNuke only displayed you on the forums when you were actually looking at the "Forum" module - but it didn't flag you as "online" in the forums display when you were browsing in the "Game" module - and most people weren't browsing the forums there, they were playing the Game module. Since 'Who is Online' there didn't reflect who was online in the Game, we didn't think about the fact that PHPBB does display it even if you were only accessing the Game page. We were trying to make a *lot* of other things work properly - that migration was a monster that took us almost two years of work to get right. So to any reader thinking to yourself right now, "you should have thought of the 'Who is Online' part of the forums" the only answer that can be given is, "yes we probably SHOULD have, but the unfortunate fact is we DIDN'T.")
We also looked at activity logs for other OS characters active around the same time and discovered Mango (pyroshroom's character) and Fenris (Khaze's character) also showed activity around these times. We knew the logs didn't show as strong a correlation with "knowing a raid was inbound" but having found what we felt was pretty clear evidence of insider knowledge, felt that it couldn't be a coincidence that they were both active for both raids around the times Southall's character was and thus they were deemed "guilty by association."
By 7/29 a majority consensus had emerged that we had sufficient information to take action. The decision to act was not unanimous, but a majority of admins felt there was enough evidence from the logs to act and that further delay in admin action would make the situation worse since it already looked like two raids had been stopped based on "insider knowledge."
The standard operating procedure for admin activity has to this point been "suspend first, then ask questions later if the user appeals" so that is what we did - a more stringent suspension for what looked like a clear pattern of "insider knowledge" and a less stringent suspension for characters deemed "guilty by association." I will personally issue an apology at this point for being the one that suggested that the smites be public to make sure it was clear that use of insider information was not okay. Plscks, who had taken the lead on investigation, administered the smites late on 7/30. Note that there was another raid by IB on OS on 7/30 after the consensus had emerged that there was enough evidence to take action but prior to the smites taking place; Southall intercepted that raid as well, but this last raid did not really factor into the decision to take discplinary action since consensus had already emerged - though the timing may have made it look like the disciplinary action was due to that particular raid instead of the two prior.
The Aftermath
Obviously, an immediate firestorm ensued when the Smites came down with unhappy players making their displeasure known on Discord.
As soon as things kicked up, the other admins engaged in discussion with the unhappy players; while this was going on, I went back and pulled additional logs to subject them to additional scrutiny because I wanted to make sure we hadn't made a mistake. I examined logs from 7/20 through 7/30 and discovered that after 7/27 the amount of intense activity of Southall watching his OS character jumped up, often for prolonged periods of time. The additional log scrutiny revealed that Southall had paying close attention to a veterinary care assistant for over an hour inside the stronghold before deciding to go invisible and step outside, which, to be quite frank, somewhat *increased* our suspiciousness of the invisibility timing. It also revealed that for the IB raid on OS on 7/30 he had been watching for a couple of hours so I reported that I was of the opinion that incident looked not like "insider knowledge" but instead like "hyper vigilance."
As conversations continued, additional information to give us context around the events we did not have at the time the decision was made came in. We learned "OS declared war on IB" before the 7/27 raid - this made heightened vigilance prior to that second raid more understandable. Most of the admins at this point still didn't feel we had a great explanation for Southall's 7/22 and 7/27 activity but also felt that the evidence might be less certain than we had earlier thought since a declaration of war would be a reason for "hyper vigilance." By this point, we had also done a more through examination of the Mango and Fenris logs and felt sweeping them up in "guilt by association" was a mistake; the suspensions were reversed.
Eventually it came to our attention that one way in which a player might use public information to watch for incoming raids was to watch the "who is online" section of the forum to see what players are active (I could be remembering wrong, because information was pouring in quickly, but I think it was actually from someone in IB pointing out to other raiders that they weren't invisible on the forums followed soon thereafter by some comments from OS about watching the forums to see who was online). As mentioned above, we simply had not thought of this as a possibility before when trying to figure out how Southall might have anticipated the raid timing (we aren't perfect, we can't and don't think of everything) and it was at this point when finally had a benign explanation for Southall going invisible. Not every admin was immediately convinced that this was the explanation but as more information came in from the OS side, this seemed increasingly likely.
As soon as that came to our attention, we realized that we had overlooked the Who is Online counter. Facepalm. The decision was quickly made that, in keeping with our changes to make it more difficult to anticipate raids based on public metadata (again, there are usually more raiders than defenders, so we feel this change generally increases the net amount of people having fun even though it makes defense more difficult), we should kill the "Who is Online" section of the forums... but unfortunately, it was shortly after this point that there was (as one might imagine) some rather heated internal discussion among the admin team about how badly this had been screwed up that led to Thonk (who had led the discussions with OS) resigning and plscks (who had taken the lead in handling the suspensions and recission of suspensions in this case) leaving as well.
I have spent my time in the hours since (in no particular order) being physically ill over this, putting in a couple of shifts at work, entertaining out-of-town family, reaching out to Southall to make sure his suspension didn't continue because things fell through cracks, consulting with Kandarin (as most know, Rincewind is currently absentee due to real life stuff) to discuss what changes need to happen and then (finally) composing this monster of a post.
So... What's going to change?
We've taken a hard look at process - right now, the process is "suspend first and ask questions later." We still think that's a good idea when violations are black-and-white (e.g., multiple characters hitting a single target or sexual harassment) because there's really no questions that need to be asked in these cases - the behavior isn't just "suspicious" - it's behavior that is "clearly against the rules."
But we also think that in situations where there's no absolute indicator, only suspicious patterns we can't come up with a legitimate explanation for as we had in this case, we should change our process to be "reach out to the player, tell them what behavior our tools have we've found and ask for an explanation first, then after hearing from the player, administer the suspension if it's warranted." The hope (for me at least) would be that at this point in the process, it becomes reformative rather than punitive - if it turns out discipline is in fact warranted, the player realizes there is an issue during the discussion and agrees with the fact that a suspension is warranted before it is imposed. In an ideal world, this would be a different admin that the one that pulled and reviewed the logs so that if there is a disagreement the admin that discussed with the player can be a more effective advocate for the "player" since he isn't defending his own work pulling logs.
I think there's also some unavoidable implicit bias in the process of having one admin "investigate a complaint" - it kind of presupposes that for someone in the community to raise an issue to us for investigation, there's likely a problem and we need to "fix" it. Hopefully the process of having different admins conduct the log pull (probably sympathetic to the accusation) and the player interview (sympathetic to the accused) helps this be more balanced.
We've also looked at how smites work - right now, the "Smite" button causes quite the public message, and that's a holdover from Nexuswar that has never been changed (I tried coding a private smite-by-default, but that's been bugged... so I think you can guess what my next coding project will be). We have realized that Smites are stigmatizing and that is not the respect we should treat our players with... even when disciplinary Smites *are* warranted, we need to make them private to protect the reputations of the players involved.
The only time Smites should be a public thing going forward is if a player explicitly asks an admin to smite them for a non-disciplinary reason (e.g., getting caught somewhere with 0 AP where AP doesn't regenerate or for dramatic/comedic effect during role-play).
A Quick Response to pyroshroom's recommendations
You are right here, and we agree with this. Hopefully the process changed outlined above where anything that has any shades of grey requires an admin to approach the player and get their side of the story first before a suspension is decided upon will drastically improve this process.1. Communication with the suspected players should really take place before punishment/smites when possible so that everyone can clarify what is happening. The initial email if it is required (for players not on discord) should contain more details about the specific infarction. In fact ‘smiting’ is really no longer an appropriate response for most player actions. This is a small player community based around people building several characters to role play, chat and compete against each other in different ways. It prizes interaction, collaboration, and communication.
As mentioned above, the plan is to make sure that "disciplinary smites" will no longer be public and are instead to be used only when players explicitly request them for other reasons, meaning "if you see a smite, that player isn't in trouble."Punitive smiting should only be used when a player is either intentionally being abusive/highly offensive, or when repeated admin/player communications have failed to resolve an issue
2. If a smite is going to be public, there should be an announcement regarding what happened and why, which has happened before in this breath. Public smites without communications should be reserved for cases with concrete evidence of multi-account abuse or blatant offensive rule breaks / cheating / alt abuse. Large games will do this for convenience but you will never see bans for things without concrete evidence.
We hope that with the changes made above, players will generally no longer feel they "need to get their case out there," since the discussions will happen before any smites or suspensions, and smites will be always be private going forward.3. Regarding Rule 7: if the players in question want to make their case public, they should be allowed to do so, trying to conceal these cases will just create rumours and drive further speculation.
This is a fair ask, and I hope that in this case, this post is public enough. I would ask in return that the community consider that the admin team is human, too, and it's one of the reasons for the last couple of bullet points in Rule #8. We are more that willing to accept constructive criticism, but it can't be stressed enough that we don't impose discipline because we like doing it, we do it because *someone* has to do it, and when we make a mistake, we would appreciate the community understanding that there is stress and emotional turmoil for us too... as I think the departures of Thonk and plscks unfortunately illustrate.4. Admin team reversing bans: It would be nice if there was a public announcement regarding the resolution of these cases that were already public in some way. A public apology would be appreciated too for the stress and emotional turmoil of the players through such an ordeal. I did receive a private apology from Plscks as you can see in my messages which I did appreciate.
And Now It's Your Turn
I've said what I feel needs to be said on this incident; hopefully this gives you some idea of what it's like "on the other side of the curtain."
Give us your feedback on the changes we're proposing and make additional suggestions for us to consider incorporating. We know it's going to take sustained effort over time and we're probably not going to get the process perfect the first time. But far too many hours have been spent on this incident and far too many people have been hurt to just wash our hands of things, make no meaningful changes, and hope things go back to the way they were.
I hope I'm not out of line by quoting some Southall's private conversation to me wherein he expressed a hope that good comes out of this and that "all parties need to bring transparency, humility, and grace to the table" so we don't have this kind of thing happen again - he's right. I hope you've found this post to bring transparency, humility and grace to the table... the floor is now yours. We're listening.
