Moderation reporting API best practices

Background

• The document answers questions regarding describing the reporting feature of SendBird.
• You can build your own in-app system for reporting and removal of objectionable content and subject when using our SDKs and API.
• We support APIs to report, moderate, and prohibit offensive and abusive messages, users, or channels.

What is the SendBird Reporting API?

SendBird’s reporting API is a system designed to allow your users to report abusive and or objectionable SendBird message content, SendBird users, or SendBird channels. The image below displays an example of where a user has clicked a message and is able to report the message.

Why would we need to use the SendBird Reporting API?

The Reporting API is for you, if you have concerns about what your users are communicating and how they are communicating it to other users. The reporting API is for you if you would like your users to have the ability to call out, and flag other users for their abusive behavior.

Furthermore, the reporting API was built to accommodate the Apple App store requirement that applications providing chat functionality should also provide abusive behaviour reporting tools. If you are building an app for the Apple Store then this API is very useful.

How does the Reporting API work?

As of this writing, the reporting API is a powerful scalable, but bare metal service. The following flow is currently possible.

1. SendBird SDK users and SendBird’s Platform API can call to report a user, message or channel.
   1. The reporting call can include both the target of the report, the reporter’s details, and also the type of abuse, and a description.
2. The reporting call triggers two actions to occur.
   1. A webhook is triggered which can be targeted at your webhook endpoint.
      1. message:report A message is reported by a user.
      2. User:report A user is reported by another user.
      3. Open_channel:report An open channel is reported by a user.
      4. Group_channel:report A group channel is reported by a user.
   2. A database entry is made to store the generated report.
3. Currently it is possible to retrieve the reports if the channel id, user id, or message id are known. This means that the current optimal method of capturing report data is to build your own database of reports by ingesting the webhook service.

How could my application implement the Reporting API?

Consider providing familiar tools for your users that are used across industries to report abuse.

• Android, iOS, JavaScript, Platform API
• Reporting Messages:
  • Consider providing little flags that can be clicked and trigger a modal for filling in report details.
  • For the report details consider providing a set menu of reporting reasons so as to minimize the users effort to make a report.
• Reporting Users:
  • Perhaps provide the ability for your user to open another user’s profile and have an interaction in there where the have a choice to “report this user”
  • As with the other reporting methods consider providing a menu of reporting options so as to reduce effort on the part of the reporting user.
• Reporting Channels:
  • In this case consider providing a flag similar to that seen on messages.
  • Again give the user an easy selection of reporting options.

What actions can be taken based on reporting actions?

SendBird Dashboard

• Once a report is received and logged through a webhook action on the report can be made by a SendBird Dashboard member with a Dashboard role that gives them access to the following where moderation actions are available:
  • Group Channels and Open channels
    • Send Admin message to whole channel
      • With or without push notifications
    • Delete channel
    • Ban users from channel
    • Mute users in the channel
    • Delete messages
  • Users
    • Deactivate a user with the ability to remove them from all joined group channels
    • Delete a user

Channel Operators: Banning users

• Single channel channel banning
  • Operators of a group channel can remove any users that behave inappropriately in the channel by using our Ban feature. Banned users are immediately expelled from a channel and allowed to join the channel again after the time period set by the operators.
  • The user can be banned indefinitely by setting the seconds attribute to -1
  • A description outlining the reason for the ban can also be included.
  • Alternatively users can remain in the channel and not be able to send a message via the mute functionality being called by a channel operator.
  • As not all users are channel operators, it might be worth considering to use the reporting functions to allow regular channel members to report bad behavior to the channel operator.
    • SendBird currently provides the availability to generate the report via SDK or platform API which ends with reporting data being made available. But, currently it would be up to the customer on how the reporting data gets handled either by channel operators, or the customer’s moderators.

• Multiple channel banning
  • Consider the case where one user continues to be a bad actor upon another user.
    • Consider in the UI offering the option for UserA to ban UserB from the current channel, but also the option to iterate all of the channels where they are the owner/operator and ban the UserB from them all.
      • Via Platform API call it is possible to Ban a user from a particular channel custom_type
    • There is also the option to globally block UserB from UserA via SDK or via Platform API.
    • The banning option could also be applied automatically when UserA creates a new channel.
    • UserA’s banned list can be fetched via Platform API or SDK.

Channel Operators: Channel freeze

• If you need all regular members to stop sending messages then consider allowing the channel owner/operator to freeze the channel. You can also download the Channel Freeze FAQ

Channel Operators: Channel Hide

• One more option if needed is to hide a channel.
• Here is the latest description:
  • Hide or archive a channel from a list of channels
  • The following code will allow you to set a channel’s status to be hidden. As a result invoking .createMyGroupChannelListQuery() will omit hidden channels in response. However, an invocations of sb.GroupChannel.getChannel() and calls to SendBird’s Platform API List channels endpoint will return hidden channels.

• To permanently delete a channel, please refer to the Platform API’s delete endpoint. SendBird dashboard admin can also delete channels.
• There are two methods associated with the visibility of a channel. All channel objects have the .hide() and .unhide() methods:

• .hide() can be invoked on any referenced channel object. The method will set the channel’s hiddenState value according to the parameters of .hide().
• .unhide() can be invoked on any referenced channel object.

groupChannel.hide(IS_HIDE_PREVIOUS_MESSAGES, IS_ALLOW_AUTO_UNHIDE, <b>new</b> GroupChannel.GroupChannelHideHandler() {
    <b>@Override</b>
    <b>public</b> <b>void</b> <b>onResult</b>(SendBirdException e) {
        <b>if</b> (e != <b>null</b>) {    <i>// Error.</i>
            <b>return</b>;
        }
    }
});

groupChannel.unhide(<b>new</b> GroupChannel.GroupChannelUnhideHandler() {
    <b>@Override</b>
    <b>public</b> <b>void</b> <b>onResult</b>(SendBirdException e) {
        <b>if</b> (e != <b>null</b>) {        <i>// Error.</i>
            <b>return</b>;
        }
    }
});

• IS_HIDE_PREVIOUS_MESSAGES: a boolean (default false). when set to true, if the channel at any point becomes unhidden – by invoking .unhide() or by automatic unhide, the channel will again appear in the .createMyGroupChannelListQuery() list. However that channel’s lastMessage parameter value will be set to null. Invocations of createPreviousMessageListQuery, and getPreviousMessagesByTimestamp will continue to return all historical messages.
• IS_ALLOW_AUTO_UNHIDE: a boolean. if set to true, after being hidden any subsequent messages sent on that channel will unhide the channel for all channel members. Messages sent from admin will also cause the channel to be unhidden. If set to false the channel will remain hidden until .unhide() is invoked on it.

How should reports be processed into actions?

• Content policy – Example
• A content policy is the guidance resource for dealing with questionable content.
  • Provide a comprehensive easy to access, and easy to understand content policy.
  • Consider providing a content policy “wall” where new users can only join the conversation once they have agreed to the content policy.
  • When taking action against bad actors refer to the content policy, violations.
  • For each item in the content policy consider how violations will be enforced.
    • At one end of violation would be a warning to a bad actor to stop their actions.
    • At the other end of the scale could be legal action and the involvement of law enforcement.
  • Where applicable, consider publishing a law enforcement guide line. This notice outlines data retention and sharing should law enforcement request it – Example

• Example: Automated moderation actions.
  • User to User Blocking
    • If more than ten Sendbird Users block (listen for blocking actions webhook) a third user, consider banning the third User from the channel.
    • If more than three Sendbird Users block (listen for blocking actions webhook) a third user, consider banning the third User from the channel for 30 minutes.
    • If a Sendbird User is blocked by users across multiple channels consider removing them from all channels and deactivating their account.
  • Message is reported
    • 10X Different users
      • When a 10th user makes a report about a message, consider automatically deleting the message.
      • Fetch the reported messages and any contextual information.
      • Have a human review the reports and decide on next action.
    • 5X Different users
      • Message the User who sent the message automatically with a warning and include the reports against them along with the content of the reported messages, matched against the content policy violations.
      • Fetch the reported message and any contextual information.
      • Have a human review the reports if the sending User contests the deactivation.
    • 3X Different users
      • Send a general warning message to the message sender with a link to your content policy.
  • Channel is reported
    • 10X Different users
      • Empty the channel of users. Review the channel contents and check for violating members.
    • 5X Different users
      • Freeze the channel.
      • Message the channel owner automatically with a warning and include the reports against the channel along with the content content policy violations that are being reported.
      • Have a human review the reports if the channel owner contests the deactivation.
    • 3X Different users
      • Send a general warning message to the channel owner with a link to your content policy.
  • User is reported
    • 10X Different users
      • When a 10th user makes a report about User A, consider automatically deactivating User A on Sendbird.
      • Fetch the reported messages and any contextual information.
      • Have a human review the reports if User A contests the deactivation.
    • 5X Different users
      • Message the user automatically with a warning and include the reports against them along with the content of the reported messages, matched against the content policy violations.
      • Fetch the reported message and any contextual information.
      • Have a human review the reports if User A contests the deactivation.
    • 3X Different users
      • Send a general warning message to the User A with a link to your content policy.