[Help] API broken. Detects "infinite loop" and shuts down even with no scripts running. Help!

Hey all - I run a pretty advanced game but the scripts I have been relying on keep getting automatically disabled because the API detects an "infinite loop" after loading 1022 translation strings to worker (I don't know what that is.)

The trick with our game is that we use the Shaped Companion Sheet (and have for almost 5 years), but others seems to be using Shaped Companion without their APIs dying. 

Can anybody help me out here? Probably somebody from Roll20, because it's NOT the scripts I've been running. It dies even when they are all disabled!

The stability has gotten worse and worse over the last 6 months and now I can't run anything whatsoever. 

Anybody got any ideas?

John

That translation strings output comes from the character sheet. Character sheet's have Sheet Workers for automaton which run in the browser but also in the Mod Sandbox. When you get the possible infinite loop detected, it's caused by the scripts not returning within some time limit. As the game grows, the startup for games can take longer and longer until the threshold gets surpassed and the sandbox just won't start. You could try moving older content to another game and removing it from the original game and see if that helps. 

For a Roll20 response, you probably want to submit a ticket from the help center (the "Send a Request" button up in the top/right of the page).

Have you tried making a copy of the game and then reloading those scripts to see if it is something specific to that particular game instance?  That error sounds like something is using language translation files, but gets stuck loading them over and over.

Edit: And The Aaron beat me to the response and may have the issue pegged.

Thanks guys. 

I submitted a ticket to Rol20 through the Help Center.

I've heard the concept of removing assets into another campaign, but that's no way to live. I have a pretty intense 4 years worth of campaign content, and as the hobby grows, so will their player base and their will be more users with "fat games." If the R20 environment can't handle that, might as well deal with that now.

Still open to any other ideas, if anybody's got one!

John

I really recommend looking into Library Games for your idle game content. You populate your Active Games with content from your Library Games while you prep a session using the Transmogrifier. It’s not a burden and it makes your Active Games much slimmer and more responsive. 

I totally agree there needs to be remediation of the root issue, and have brought it up to the devs again (and linked this thread), but in the short term, I'm hoping that archiving can save your current gaming experience. 

I get the computational efficiency of working only from stuff that's loaded locally. But if my players suddenly say they want to go see this or that NPC, it CANNOT BE that I had to have thought of that beforehand and have manually dragged it over in the Transmogrifier for every session or adventure. 

And then wouldn't you have version-control issues? So anything you changed on the active game you'd have to remember and copy back the updated version to the Library Game?

I literally cannot conceive of doing all that extra, mindless work to game. I would quit gaming before I did that.

Can it really be true that my game has just outgrown Roll20's technical capacity, so now I'm losing more necessary functionality and time-savers? I might be in a death spiral. 

What am I missing? I feel like I'm taking crazy pills. I know I'm at the top end of GMs in pushing the system, but I didn't think I was THAT far out of the mainstream. Has this happened to hundreds of GMs already and I just didn't realize everyone was coping with it?

Really appreciate your guys' help. Can you just confirm—do you really keep most of your stuff in a separate game and try to guess what you'll need for each session/adventure?

Yep. And I make semi regular backups of the entire game in its current state. I've lost serious data in the past due to some communication glitches.

Number of characters shouldn't be that much of an issue anymore, at least not for game loading, due to the new Lazy Loading feature. Total number of tokens in play is still a deal. But yeah, try duplicating the game and see if you see a performance increase. Sometimes it can just get rid of accumulated cruft.

I guess a lot boils down to how you run your games.  For npcs in my games, most don't end up getting a character sheet or a token.  Instead, they are a handout with some character art, likely with the handout being editable by players so that they can put notes on that npc there if they wish.  I only create sheets/tokens for npcs that I really expect to enter combat and need a full slate of abilities.  If an npc just needs a roll or 2 (like merchants needing some sort of appraise or lore rolls), then I often put bonuses to certain rolls in the GM notes area, or build the roll(s) as text links in the handout. In the case of npcs with actual character sheets, I don't usually move them out unless I feel it is unlikely the characters will be interacting with them again at all or for a very long time.

I use multiple library games.  Some are game system specific like Pathfinder Character Library or Palladium Bestiary, while others are more universal like City Buildings Library or Underground Maps Library.  This is mainly for organization to make finding things in the Transmogrifier list easier.  Maps are usually (for me anyway) the easiest bits to anticipate the need for.  They are also pretty quick to drop in and use as long as you have a decent naming scheme for your maps.  I have no idea how lazy loading has impacted maps, but they certainly add up in the overall data "size" of your game.

As far as the "I can't be the first" comment, you aren't as far as game size goes.  What I have seen a few times in the past cross the forums was that after a game crosses a data "size" threshold (which I am sure changes as technology advances), then certain functions stopped working like copying a game.  At that point, the game was so large that the Amazon servers were timing out during the copy.  That meant that rollbacks no longer worked either, because the automatic copies the the rollback feature relies on had been failing.  If you are able to copy your game, then you should be free from that particular issue.

That's good practice. I run a campaign based in a small village with about a hundred named characters. The vast majority of them are "mooks", all running off the Commoner sheet, but with different art and names, and some other specific data hidden in token notes and bars. I only use separate sheets for important NPCs that need abilities or are modeled after character classes.

Holy smokes, guys. This is just not feasible. It looks like in order to do this, I will have to transmogrify over hundreds of pages and characters, one by one... and then somehow remember all their names and go into my playable game to delete them each out. Individually. (And for maps I'll have to Restore them from the Archive, one by one, then go to that map in the fiddly selector and Delete each one.)

I would have to shut down my game for weeks to find time to do that. I cannot believe everyone keeps their entire campaigns in an entirely separate "game" because Roll20 literally cannot hold a long-running campaign and still parse the objects. It boggles my mind. It feels like this is an episode of Twilight Zone and everybody's telling me that society has always been cannibals, didn't you know? 若

Anyhow, this looks like the end of my 10 year journey on Roll20. After 10 years and thousands of hours and dollars spent, the product is a failure (for me). It seems it simply cannot handle normal operation for a long-running campaign without this mind-boggling (and for me, completely infeasible) workaround. I'm in utter shock. 

Thanks to all of you for trying to help. You guys have been wonderful and altruistic contributors to the community and the product, enabling lots of good times for lots of people. 

Before you leave...
- Foundryvtt:. Best practice #1:  Keep your play game small and archive your pages, actors and journal items to a compendium. Which amounts to exactly the same as Library Games  and Transmogrify. And if you would play a 10 year campaign, your data would have died 10 times over due to the many breaking changes inbetween releases.
- Fantasy Grounds would just crash before reaching this amount of pages, you will never be able to create this amount of pages and neither play a 10 year campaign.
- Owlbear Rodee: has basically one map, you will never be able to create this amount of pages and neither play a 10 year campaign with this amount of assets.
- Tale Spire, Shard Tabletop, all of the others VTTs would struggle with the shown amount of maps.

I've cleaned up long running campaigns before, as i run two of them. About every half year i just make a copy of the game and spend half an hour to delete all stuff that was not used.

Only Roll20 will allow you to create a game with the Curse of Strahd module, then transmogrify in all assts of Tomb of Annihilation (including that hex reveal map), then add a dozen Kobold Press GameAddons. It will be slow like a overloaded mule, it will look like Rocky Balboa in round 7... but its the only vtt that will still shakily stand after taking so much battering.

Martijn S. said:
- Foundryvtt:. Best practice #1:  Keep your play game small and archive your pages, actors and journal items to a compendium. Which amounts to exactly the same as Library Games  and Transmogrify. And if you would play a 10 year campaign, your data would have died 10 times over due to the many breaking changes inbetween releases.

Probably fair to point out that updates aren't forced in FVTT though, they're manual :) . Nothing is stopping you from keeping a stable version for 10 years. Your point still stands though - my mind boggled at the size of Mad Mage's JSON on Roll20, and that doesn't break the VTT.

Oosh said:

Probably fair to point out that updates aren't forced in FVTT though, they're manual :) . Nothing is stopping you from keeping a stable version for 10 years. Your point still stands though - my mind boggled at the size of Mad Mage's JSON on Roll20, and that doesn't break the VTT.

For comparison, here are !Survey reports for DotMM and my homebrew campaign (which despite using a Library Campaign is still very large).

My personal campaign is bigger in many metrics. Probably the biggest difference is that a lot of my pages are far simpler than a typical DotMM level. I just ran these reports on my Chromebook this morning; both campaigns loaded lickety-split. In general, it's not the oomph of the hardware that is the bottleneck. That being said, I'm sure I would run into performance issues on either game during actual play. But simply loading and moving things around shouldn't be an issue that is affected by level of hardware. Especially not with Lazy Loading.

As for the process of maintaining a Library Campaign, it can be quite daunting to set one up, but there are some practices to help:

• Create a fresh instance of DotMM. Now you have a base library campaign, ready to use.
• You can simply delete anything you don't need that you have not modified from the live campaign. Deletion is much easier, as you can mass delete a lot of stuff that is in folders.
• As you make modifications in your live campaign, transmogrify them to the library campaign.
• If you keep on top of it, maintaining a library campaign is less of a chore.
• Make periodic copies of your game. If you screw up something from the library copy, you have that backup you made a few weeks ago that might have it. (put the backup date in the title). The number of games you have on your account has no limit.

I still don't understand the issue that is affecting your APIs. The arbitrary number of 19 scripts in particular. I have a strong feeling there is something more going on there in one of the 19.

I have two campaigns currently running; Waterdeep:Dragon Heist and the Alexandrian Remix of Waterdeep:Dragon Heist. Both campaigns use the same Library and I also have a Development Campaign using the same Library that I use to develop content, (I have an extensive home brew Sewers of Waterdeep set of maps that are built with tile sets, also some of the canned maps for towers and buildings I have reworked with tiles). It all seems to work beautifully. If the players head off to an unexpected location I simply transmogrify in a pre-populated map and away we go. When the players are using a map I move it to near the top of the page selections, the ones that are left at the bottom I delete from time to time because they are not being used and I have a quick backup in the library.

Just to close the loop here, I got a very nice message from Miles at Roll20. TL;DR is that Roll20 cannot handle a game of this size.

Hi there,
Thanks for reaching out and sorry for the trouble on this! I took some time to read through the forum thread you mentioned and our dev team is also aware of what’s going on here. I can say confidently we have a bug ticket in place for the larger conversation regarding the API such as the time out and other potential improvements.That said, I’ve done some more digging on our backend to see what I could find. Looking at the sheer size of the game, you have broken my previous record of game sizes I’ve seen in my 5 years of doing this. This game is absolutely massive, just to put it in perspective your game is roughly the size of 5-6 copies of Dungeon of the Made Mage put together (our largest out of the box module). This puts you in the top 0.0001% of games by size.All of this to say, I think you have managed to push the limits to their breaking point. At this stage, I don’t believe there is anything myself or our team can do to fix this game directly without those changes to the API which are not guaranteed to fully resolve the issue in this case (you’d more than likely return to the degrading performance you experienced over the last 6 months).

At this point, I'm simply not willing to do what amounts to manual memory management and version control in my head. Don't have the bandwidth. 

We'll play without the API, which will make the game even more of a pain, until I can find a time to wrap it up. 
I don't blame Roll20 for for not having the capacity to run a campaign like mine. I realize it's an outlier. It's just too bad.

It is what it is.

Thanks for all the help and advice everyone!

Shoot, sorry to hear that, John. OTOH, it sounds like you have an extremely detailed and long-running campaign.

Campaign's been great! Started as West Marches based on XCOM, with homebrewed crafting, base-building, research, etc. Multiple characters per player. Super low magic, but magic is especially powerful as a result. Character death has run about 1 per 12 sessions (but again, each player has 4-6 at any one time). Money is SUPER valuable. Recently been running a politics mini-campaign inside the larger campaign where they try to install their chosen lord into place instead of a usurper, and have to win the favor of the local vassals. 

Very advanced games are fun!