AI Moderation

Overview

AI Moderation in the CometChat SDK helps ensure that your chat application remains safe and compliant by automatically reviewing messages for inappropriate content. This feature leverages AI to moderate messages in real-time, reducing manual intervention and improving user experience.

For a broader understanding of moderation features, see the Moderation Overview.

What Triggers Moderation?

Moderation is triggered only for the following message types:

• Text Messages
• Image Messages
• Video Messages

Other message types are not subject to AI moderation.

Handling Message Moderation Status

When sending text, image, or video messages, the moderation process introduces the following changes to the message flow:

1. Initial Status:

   • On successfully sending a text, image, or video message, the response will include the moderation status as pending.
   • You can fetch this status using the getModerationStatus() method, which returns a string pending.

2. Moderation Result Event:

   • Once the moderation service processes the message, a real-time event is triggered in your MessageListener as onMessageModerated.
   • This event provides the message object with the updated moderation status, which will be either approved or disapproved.

Example:

Swift

// Check moderation status after sending a message. This check is only applicable for TextMessage & MediaMessage class.
if let message = sentMessage as? TextMessage {
    if message.getModerationStatus() == "pending" {
		// Message is under moderation
    }
} else if let message = sentMessage as? MediaMessage {
    if message.getModerationStatus() == "pending" {
		// Message is under moderation
    }
}

// Listen for moderation result.
func onMessageModerated(moderatedMessage: BaseMessage) {
	if let message = moderatedMessage as? TextMessage {
	    if message.getModerationStatus() == "approved" {
	        // Message approved
	    } else if message.getModerationStatus() == "disapproved" {
	        // Message disapproved
	    }
	} else if let message = moderatedMessage as? MediaMessage {
	    if message.getModerationStatus() == "approved" {
	        // Message approved
	    } else if message.getModerationStatus() == "disapproved" {
	        // Message disapproved
	    }
	}
}

Objective-C

// Check moderation status after sending a message. This check is only applicable for TextMessage & MediaMessage class.
if ([sentMessage isKindOfClass:[TextMessage class]]) {
    TextMessage *message = (TextMessage *)sentMessage;
    if ([[message getModerationStatus] isEqualToString:@"pending"]) {
		// Message is under moderation
	}
} else if ([sentMessage isKindOfClass:[MediaMessage class]]) {
    MediaMessage *message = (MediaMessage *)sentMessage;
    if ([[message getModerationStatus] isEqualToString:@"pending"]) {
		// Message is under moderation
	}
}

// Listen for moderation result
- (void)onMessageModerated:(BaseMessage *)message {
	if ([message isKindOfClass:[TextMessage class]]) {
		TextMessage *textMessage = (TextMessage *)message;
		if ([[textMessage getModerationStatus] isEqualToString:@"approved"]) {
			// Message approved
		} else if ([[textMessage getModerationStatus] isEqualToString:@"disapproved"]) {
			// Message disapproved
		}
	} else if ([message isKindOfClass:[MediaMessage class]]) {
		MediaMessage *mediaMessage = (MediaMessage *)message;
		if ([[mediaMessage getModerationStatus] isEqualToString:@"approved"]) {
			// Message approved
		} else if ([[mediaMessage getModerationStatus] isEqualToString:@"disapproved"]) {
			// Message disapproved
		}
	}
}