There is a lot of activities over at Roblox HQ lately on platform security. Today they have announced another big investment in its multi-layered approach: real-time multimodal moderation.
Roblox had not have the best of reputations in the past when it comes to platform safety. But based on their actions, there are some seriously concrete steps actions. Not talked about – actioned.
Proactive Safety: Scaling to 100% Playtime Monitoring
Part of the magic of Roblox is the ever-expanding and changing content produced by creators. At any moment, creators are publishing game updates and adding new badges or pets, and users are changing their avatars’ outfits and painting or building things in real time. These are all reviewed before it’s published to the platform and remove anything that’s against our guidelines.
Multiple layers of moderation tools catch the vast majority of problematic content on Roblox, and anything that goes against the Community Standards is rejected. But it also leverage user reports to help find anything that may have missed and take action accordingly.
The dynamic nature of Roblox games means content changes continually based on how users combine previously approved avatars, clothing, and movements. For example, in games that allow free-form drawing, a user could draw an offensive symbol or item. That’s why they recently launched a new AI system for real-time multimodal moderation that can scan these combinations together.
Beyond Single Objects: The Power of Scene Evaluation
Traditional AI moderation systems are designed to evaluate one object at a time and often lack context, missing combinations that could be problematic in ways that the individual items aren’t. Our new real-time multimodal moderation system evaluates an entire scene – including avatars, text, and 3D objects. It captures all of these elements together in a specific moment and assesses whether the full scene breaks the rules. If this type of problematic behaviour repeatedly occurs in a single game instance, the system will shut down just that instance (also called a server), rather than the entire game.
Since launching this multimodal system, approximately 5,000 instances per day that violate the Community Standards has been shut down. As they train and scale, they’re constantly improving the accuracy and working with the community to minimise false positives.
This multimodal system is being scaled up to capture and monitor 100% of playtime. But there will always be individuals working to circumvent any system, so technology is actively being developed that goes beyond shutting down servers. They are working on ways to identify specific bad actors so they can remove them without disrupting the experience for well-intentioned players.
Enhanced Transparency: New Tools for Roblox Creators
Creators are provided with greater transparency into the results of this multimodal moderation system with an addition to the safety overview dashboard. As noted above, game servers are shut down when they’re overtaken by bad user behaviour. To give creators more visibility into how often this is happening in their games, a new chart has been added to their existing Creator Dashboard.
Creators can now see how many of their game servers have been shut down for bad user behaviour (i.e., breaks harassment and discrimination or romantic and sexual content policies). This helps them identify a sudden increase so they can act before shutdowns affect their broader community. They can then take a closer look at their game and decide whether changes are needed to custom emotes, avatar editing tools, or in-game user creation features to help prevent problematic creations.
Industry Collaboration: The DLC Leadership Certification
Roblox, Keyword Studios, and Riot Games are partnering with research psychologist and Games for Change Research Director Rachel Kowert, on a new certification program for digital community leaders. Roblox will contribute expertise on community moderation and prosocial design to pilot and help shape the curriculum for the new DLC Leadership Program. The goal of the initiative is to address a lack of standardized training for online moderators, community managers, and creators in the gaming space. In a recent article, Kowert said the program aims to “translate research on gaming communities and online behaviour into practical tools that digital leaders can use to build more resilient and sustainable online communities.”
By participating in this effort, we’ll help lead the industry, developing a first-of-its-kind standardised certification program designed to benefit the Roblox creator community and the broader gaming industry. Once completed, the program will help creators learn the critical skills required to effectively moderate and manage their own growing communities. Roblox believe that training more moderators and creators on the best practices for healthy, respectful online communities will help keep online gaming more positive for everyone.
