A Note from the CTO, Mike Todd: Stability, Accountability, and Our Path Forward

Hi everyone, I’m Mike Todd, formerly the CTO of DriveThruRPG and now, as of this past December, the CTO of Roll20. As a long-time TTRPG player and an engineer, I know that when you sit down for a session, the tech should stay out of the way. You're there to play a game, tell a story, and have fun with friends. Not to troubleshoot a VTT. Lately, we haven’t been meeting that standard. Recently, we’ve had a few incidents that have caused instability for some of you. I want to be open with you all about what’s happening behind the screen and how we’re fixing it. The Perfect Storm The experience has been less than ideal recently, and we know that the frustration has landed squarely on you. Some of the issues we’ve seen were triggered by instability in external services like Cloudflare (the service that serves images in the VTT) and Firebase (one of our primary database services), but the truth is that we should have been better prepared to deal with those realities. Relying on third-party infrastructure does not absolve us of our responsibility to you. In fact, it raises that bar.  Infrastructure & Stability : To put it bluntly, Cloudflare has been less stable than we need it to be, evidenced by the global outage in November that impacted almost all of the Internet. We’ve seen continued issues with their service even after that, and we are evaluating options to switch to a different, more stable provider for this part of our infrastructure. We are also actively researching alternatives to Firebase to further harden our architecture. The January Rush : I think we can agree that growth is great for our hobby, but that added strain puts every tech "bottleneck" under a magnifying glass. This month, those bottlenecks were put to the test because this is the busiest January we’ve had in years. Owning Our Issues Yes, there were some external issues, but I have to say we’ve had some misses that were entirely on us. One example is that we released the new D&D sheet in a buggy state. Last January we spent over a month in a laser-focused "bug-squishing" mode, which fixed over 500 bugs and made the sheet a lot more stable. Our team has worked hard to make this a better experience for everyone, and that hard work has paid off. But while the new D&D sheet is in a much better place, there are still some smaller bugs remaining, as well as one BBEG: intermittent issues when multiple people have the same sheet open at once. This is a complex concurrency challenge, and it is the top priority for our back-end engineers right now. Much more recently (this very week), we identified a wide-ranging issue, which has been the team’s primary focus this week. If I can lapse into tech speak for a moment, we noticed a memory usage creep on our web servers (Kubernetes pods, for the geeks out there) that was causing some of those instances to go into swap. This created a frustrating experience for some users that was often intermittent: You might have had a laggy session while your friend in the same game felt nothing, or one page load might have timed out while the next was nearly instantaneous. It was a "luck of the draw" issue based on which of Roll20’s server instances you hit.  My Infrastructure Philosophy Whenever something in our infrastructure breaks, I have a standard a three-phase response: Fix it : Put out the immediate fire. Instrument it : Set up monitoring so we know before it happens again. Automate it : Build self-healing measures so the system corrects itself without human intervention. The Road Ahead At times internal bugs and external outages happen concurrently, making them a nightmare to disentangle. But we have to admit that, regardless of the source of the problem, the result is the same: your game night was interrupted, and ultimately that’s our responsibility . If Cloudflare or other services are unreliable, then it’s on us to find a way to make them work or move to another service that is more reliable. In addition, we need to ensure all aspects of our systems can detect and alleviate those problems when they arise, so that your experience is not degraded. Now that we have identified and addressed the primary cause of that memory usage creep, we are seeing immediate results: reports of “server 500” errors (a specific type of error), image loading failures, and spontaneous logouts have dropped significantly. We also have many reports of people saying things are working now, that weren’t working a few days ago. But we aren't stopping there. In addition to keeping a close eye on things over this weekend to make sure your games run smoothly, here are our action items for the coming weeks to ensure this stability sticks: Hardening Infrastructure : We are working directly with Cloudflare engineers as they investigate the recent instability on their end. And we are investigating the possibility of moving that infrastructure back to AWS (Amazon Web Services). Active Monitoring & Auto-Healing : We are in the process of adding layers of additional monitoring and "auto-healing" protocols. Our goal is for the system to detect and fix issues before you notice something is wrong. “WebGL Context Lost” Investigation : This is an error some people were experiencing in the VTT which we believe is resolved by the Kubernetes fixes, but we are still keeping alert in case more reports come in. Firebase Alternatives : We are actively researching alternatives to Firebase. I know we've fallen short, and we are committed to doing better and being transparent with you as we navigate these challenges. If you’ve been affected by these issues, then I apologize to you and hope you can give us some time to make this right. We owe it to you. Thanks for being part of this community, and for sticking with us as we work through these problems and continue striving to be a better partner for your games. Sincerely, Mike Todd CTO

Hultz (Keeper of Secrets) said: I play 5e exclusively. I want to upgrade from 2014 to 2024 but am afraid that things won't work such as some of my api scripts and some of my macros. It seems like every time I log on there is a different issue, mostly syncing with macros and scripts. Also there are a lot of duplicate die rolls. It is disturbing when a weapon attack rolls the same number three times in a row. Once in a while yes, but over and over again. (Shouldn't there be a tag of some sort on the die roll indicating which roll is being used?)       Please stabilize the platform.  Thanks for the feedback! Let me address your concerns: Macros & API Scripts.  We've done a lot of work to make macros compatible across 2014 and 2024 sheets. Most common macros should work on both, but there are some attribute differences due to how the new sheet better breaks down all the elements that feed a stat using integrants.  API/Mod scripts are still in progress, one at a time, with priority on the most popular ones like ChatSetAttr that other scripts and macros often rely on. If you have specific macros or scripts that break when switching, please share them and I can help troubleshoot or escalate issues we need to fix. Duplicate Die Rolls.  When you say "duplicate die rolls" are you seeing: the same visual number appearing multiple times in one roll output (like seeing "14" three times when rolling 1d20)? Or the same result appearing across multiple separate rolls (rolling 1d20 three times and getting 14 each time)?

Plural and Greg - For Demiplane, I would appreciate hearing any specifics you have on what is most annoying to you about the integration. We are currently working on two things that would improve this: 1) We fix an issue with character sheets not caching in the VTT leading to long load times each time a character sheet is opened. This has (we hope) largely solved the issue of character sheets opening slowly after the first time they have been opened in a session. Now we are working on a 2-stage project to improve the initial, first opening of the character sheet - to get this to be much faster. If this is one of your areas of annoyance, we are working on it right now. 2) We are also currently working on improving the cross-platform authorization to get some of the annoying log-in requirements out of the way. Besides these two things, what are the biggest pain points? Steve Wieck President Roll20

I really appreciate all the feedback, and also that others here in the community (and Roll20) have responded to much of it! Keeping with the theme of transparency, I had some tragedy in my personal life last month, and haven’t responded to this thread as I should have. I wanted to make sure to give these questions and comments the full attention and thoughtfulness they deserve. A lot of these comments are not so much about system-wide stability, but specific bugs or features. I know that from a customer perspective, Roll20 seems like a monolith. But we are made of several different teams, each with several types of people on them — we have product managers*, designers, developers, QA people, content managers, marketing people, finance people, etc. My sphere of influence as CTO is mostly in the development and QA portions of that, but I’ll do my best to speak to those other parts as well! Also as a note, within the next week or so I am going to be making a post centered on performance updates, which will also touch on some stability issues. So keep a lookout for that! * I know this term might not be familiar to those not in software development, so as a basic definition that I’m taking from Atlassian : a "product manager" defines product strategy, roadmap, and features, balancing business, technology, and user needs.

Mark G. said: While I appreciate the desire for stability, I would suggest starting internally. The sudden release of the page design and new menu structure was annoying and off-putting. I relied on the "new messages" icon at the top of my game page, and I was late seeing new messages because it had been moved. I eventually found it in a location I think the least intuitive for any long-time user: at the bottom of a new pop-out menu panel, isolated from the menu links in a mostly-empty panel.  This is one of those “product”/design oriented questions, but I’ll give my perspective on it. I definitely understand how frustrating it can be when things move around – especially in a way that’s unintuitive! Your specific feedback here makes a lot of sense, and I believe our head designer Brittany is going to be weighing in. As a more general statement, we know that any change (especially changes that move elements from one place to another) can be jarring. Our design team does vet new designs with the community before we release them, and if you’d like to be involved in that then you can use this link to join our Discord server, and use our Server Guide to join our Lab Rats program.

Ulti said: I don't use Demiplane, and neither do my players. But since the Demiplane integration, I cannot transmogrify the characters created by my players to test games. The message says "Error copying Demiplane character". Other have the same issue, like  in this thread  or  that thread . Ulti, Thanks. I've asked our platform team to get an update to that thread. I know that Miles discussed the issue with them, but I'm not sure where they are on a fix, so expect an update in one of those threads. To the best of my knowledge, that issue is not actually related to Demiplane per se, it's related to work done for Roll20 Characters to give players (and later GMs) easier access to character sheets outside of the VTT. The way that was originally architected had some knock-on effects causing issues like this one. Platform team has been reworking some of that architecture, but I'll need to let them speak to this specific issue. Steve

Dr DM said: But, as always, these reports will fall on deaf ears. I certainly wish that we could snap our fingers and magically make every part of our platform instantly do all of the things we would like it to do. The most frustrating part of our job is prioritizing what order to do the things that we know would improve the platform, because that prioritization is always this Sophie's Choice of which of the things we really want to do will get worked on now and which ones have to wait in line. There is a difference however between 1) us not listening and having deaf ears and 2) us not being genies who can magically make a wish and have the platform(s) do everything we all want them to do instantly. When we seek out feedback about what's "most annoying", it's not because we are blind to the fact that parts of the integration are annoying, or because we've been deaf to prior feedback and requests. We know it's not perfect. We know parts of it are annoying. We play games on it ourselves every week. It's because we're looking for current takes that help us prioritize among the annoying things to help us guide the prioritization decisions. This gets further complicated because when we're prioritizing, we're using information that is not always apparent to the Roll20 community, specifically what amount of engineering effort is estimated to fix each issue (or add each feature) and when there are dependencies that certain things are logically done first before we can then do a fix or feature. So if Annoying Thing A is 2x more annoying than Annoying Thing B, but A takes 10x the engineering time to fix than B, we're probably going to do B first to most efficiently remove the most annoyance as soon as possible. If we're not communicating often enough about this, it can leave the community scratching their heads wondering why those morons at Roll20 fixed B before fixing A, when A is clearly the worse problem. As I said, right now, we're fixing cross-platform login and sheet loading performance because we think those are the two worst problems that we can more efficiently solve first. I wasn't asking for a recitation of every other annoying thing about the integration, I was asking for the "most annoying". I'm asking for your view on that tough prioritization decision of what to fix next. Would you care to look at your list of annoying things and pick out the 1 or 2 things on the list that are most annoying that you think should be fixed next in line? Steve

Hultz (Keeper of Secrets) said: I play 5e exclusively. I want to upgrade from 2014 to 2024 but am afraid that things won't work such as some of my api scripts and some of my macros. It seems like every time I log on there is a different issue, mostly syncing with macros and scripts. Also there are a lot of duplicate die rolls. It is disturbing when a weapon attack rolls the same number three times in a row. Once in a while yes, but over and over again. (Shouldn't there be a tag of some sort on the die roll indicating which roll is being used?)       Please stabilize the platform.  Dean responded to this, but just to reiterate: We have been putting in a lot of work to make sure that macros work on the new sheet, and we’ve upgraded a lot of the top mod scripts as well. We’ve talked about that in some of our blog posts, including this one .  As far as one click causing multiple rolls, could you write into customer service with some details about that, and DM me a link to the ticket so I can make sure it’s followed up on? Some examples of good information to have: Are there specific attacks and/or sheets that this tends to happen with? If so, which ones? (Giving customer service a link to the game in question would help, too.) Does it always happen, or is it sporadic? Is this in the VTT, Roll20 Characters, or both? Do multiple people have the sheet open at the same time (e.g. DM and player)? I’ll note that there is a purposeful exception here for some spells like Eldritch Blast and Magic Missile — clicking on that once is supposed to roll multiple times, because (according to Rules As Written) it triggers multiple separate attacks or damage rolls when you use it.

Ken S. said: 1. You need to do a better job supporting non-D&D games.  Yes D&D is your bread & butter.  But I recently started playing Mongoose Traveller in addition to 5E (2014) and I'll just say that you need to hold your vendors to a higher standard for things like the compendium.  And, somehow, you need to get them to update their sheets to the new technology you introduced with the 2024 D&D sheet. I realize its volunteer-supported, and I'm not disparaging those folks, but players buy games: I bought a shelf of Mongoose's hardcovers, in part because only a small subset of their books were available through Roll20's compendium, and those potential customers should give you some leverage with other publishers to do a better job, but you'd need to drive that, or make tools to support conversion or something, and maybe find a way to fund some of it from sales of their rules/adventures/etc; it's obvious they won't do it on their own). Taking this in three chunks for ease of responding, and also because the editor isn't letting me un-quote in the middle of a quote. :) One of the great things about our site is that publishers, third-party converters, and the community have the ability to convert their own titles and make their own sheets. Our teams are always chatting about ways to improve the conversion experience and making it easier for publishers to create within our tools. We are aware that the current process is not the best it can be, but those conversations are top of mind and continue among our developers. These updates can take some time due to the technology, the bandwidth of the team, and the roadmaps each team is committed to. If there is an issue with a conversion, our Partner Relations team is ready to work with these converters to try and make the experience the best it can be. After chatting with Partner Relations (who works with our partners and their conversion teams directly), we can confirm that the conversion team for Mongoose especially are extremely communicative and love hearing feedback to improve the experience. More titles are coming out from Mongoose in the future on Roll20 (as well as the Traveller NEXUS on Demiplane), so your feedback would be extremely helpful in shaping the current titles. Todd, one of our Partner Relations Leads, said he’s more than happy to take that information over to them. If you could submit a ticket to our Help Center , they’ll get it to Todd who will work with Mongoose.

Ken S. said: 2. Can you please, please, please, find a way to get the 2024 sheet to work like the 2014 sheet (and character sheets in D&D since the 1970's): multiple pages (tabs), lots of room for the essentials: spells, equipment and attacks (high-level characters feel like they're wearing a straitjacket), and none of this "half the sheet is wasted white space" and tabbing out 1/4 of the page.  Yes Wizards' did it that way rather than mirroring the paper sheet in their rulebook. Does that mean it was a good idea? No. Screen real-estate is precious, and maybe 8.5x11 pages aren't the answer, but things that pack less info per square inch certainly aren't the answer either. This is in my sphere of influence! Not since the beginning, but for the last year or so I’ve had a key role in approving what gets done on the new D&D sheet (which I also want to point out is usable for 2014-only characters as well). This is a pretty big subject overall, but it seems that a lot of your response comes down to wanting more information density — seeing more things at once without having to scroll around. That is something that we’ve been striving to do, but it’s definitely a process that we do in chunks. If you have any specific feedback, I’d love to hear it though! Either here, or I’m more than happy to open up a direct dialog with you. As just one example, here's how we've compacted Abilities and Saving throws in the original vs. current version. Original: Current: We do also have to make sure things are far enough away from each other to prevent misclicks, and also keep in mind touch interfaces for people using the sheets on phones or tablets through Roll20 Characters.

DM said: Steve W. said: Thanks. I've asked our platform team to get an update to that thread. I know that Miles discussed the issue with them, but I'm not sure where they are on a fix, so expect an update in one of those threads. . . FWIW, Miles didn't discuss the issue with 'them' as I am the author of one of the two threads and had no interaction with Miles at all.  I got a (seemingly) canned AI response to my help ticket indicating that someone or someone-s (or something) calling itself/themselves 'The Roll20 Team' couldn't reproduce the issue. In essence, the forum community was faster and more helpful than the so-called 'Help Desk' and, in my opinion, that can only further erode trust and goodwill toward Roll 20 -- particularly as that community grows more and more frustrated. I think there's a miscommunication here. When I said Miles discussed it with "them", my intended antecedent to "them" was "our platform team" not (as I am inferring you read) "that thread" (by which by extension, it would mean author/community members in that thread). Miles handled a Help Desk ticket on the topic, confirmed the problem, and took the problem our internal platform team to get it on their roadmap to solve. That's how a Help Desk inquiry of this nature ought to go. I'll make sure customer service team sees your note that you got a decidedly different experience on your own Help Desk inquiry. Apologies that was handled so poorly. Steve

Ken S. said: 3. And please give us the stuff that people have been asking for since the dawn of roll20 (you have made inroads there recently), I'm particularly asking for a ways to homebrew / customize / overlay compendium info.  Few people I know play absolutely by-the-book, we all have custom magic items, special equipment, and other things, and we'd like a way to have our version show up for players instead of the standard one, or to add the items unique to our campaign.  This is likely more importand for long-term tables than for the commerical for-pay DMs, but we matter too. We’ve done a ton with homebrew in the new sheet (including letting you create your own custom classes and subclasses!). But I’m also aware that a full “home brewed compendium” feature has been one of the top things on the community’s want list for quite a while. The honest answer here is that we’ve scoped some of the work, and a custom compendium is a truly massive project. If we tried to do it “in one go,” then you’d see no movement on it (and other projects) for several years until finally it would all come out at once… and potentially be done in a way that a lot of people don’t like. As you can see even in this thread, different people have different needs and expectations for that sort of a system. So instead, we’ve been doing it in bits and pieces and adjusting as we go based on feedback from users like you. We’ve done a lot with homebrew spells and equipment, and as I mentioned above we’ve implemented custom classes and subclasses (including automatically giving class features at certain levels). You can also use the Transmogrifier by itself or in combination with these to copy your homebrewed stuff between games so you don’t have to create it from scratch each time. We’ve created the Shop Sheet as well, and we plan to allow that to directly transfer items to characters, which will be able to sort-of serve as a mini custom compendium. Again, I realize that this isn’t the full custom compendium experience you, and we, and many others all want – but when we do decide to take on that massive project, we'll use the learnings and community feedback from these other steps to inform our approach.

Ken S. said: Oh, and a bit more openness on the roadmap (like a rolling 12-month forecast of changes planned) and a heads-up on "we're going to change this important thing next Friday" (and maybe not making major changes a day before a gaming weeked) would all be appreciated. That’s something we’ve been thinking about a lot internally, especially in terms of transparency (this post of mine kicked off a lot of internal discussion as well). In the time since you’ve given this feedback, we’ve released our public roadmaps !

Mike T. said: Mark G. said: While I appreciate the desire for stability, I would suggest starting internally. The sudden release of the page design and new menu structure was annoying and off-putting. I relied on the "new messages" icon at the top of my game page, and I was late seeing new messages because it had been moved. I eventually found it in a location I think the least intuitive for any long-time user: at the bottom of a new pop-out menu panel, isolated from the menu links in a mostly-empty panel.  This is one of those “product”/design oriented questions, but I’ll give my perspective on it. I definitely understand how frustrating it can be when things move around – especially in a way that’s unintuitive! Your specific feedback here makes a lot of sense, and I believe our head designer Brittany is going to be weighing in. As a more general statement, we know that any change (especially changes that move elements from one place to another) can be jarring. Our design team does vet new designs with the community before we release them, and if you’d like to be involved in that then you can use this link to join our Discord server, and use our Server Guide to join our Lab Rats program. Hello Mark! Brittany here from the Design team. I totally understand your frustrations about the navigation changes! As Mike said, we do run our designs by our users and I encourage you to join us on Discord. As for the specific changes you mentioned about the Messages location, we actually noticed the placement issue and already made improvements to it. You can check it out now and please let us know if you have any other concerns. We love hearing from users like yourself. Thanks

Gold said: Steve W. said: Steve Wieck President Roll20 What is the number of Votes needed on the Suggestions forum for the thread to get a response from Roll20 Team?&nbsp; There are many Suggestions now with in-excess of +200 votes, but no reply from Roll20 Team. In the past Roll20 said that 200 votes would garner a response.&nbsp; <a href="https://app.roll20.net/forum/category/385189/?orderby=popular" rel="nofollow">https://app.roll20.net/forum/category/385189/?orderby=popular</a> I think some of the useful customer feedback is there, some longstanding user frustration points.&nbsp; Since 2022, we've been keeping with the process outlined in the forum's pinned post. "We pay particular attention to suggestions that have gained support quickly, those that are small quality-of-life enhancements, and those in the top 20 of all suggestions. If a suggestion is in the top 10, you can expect updates from staff in thread when the suggestion changes status." For example, today we were able to close this one asking for Copy + Paste for Doors &nbsp;because it was in an area that the team was working already to support the Map Pins suggestion, which was at No. 2 late last month when it was closed. My stats are a little bit old, but since 2022, we've completed 20 of the Top 50 Suggestions by Vote Count (representing more than 10,500 votes).