API Inconsistently Responsive

Oh wait, that's me, lol! I can only report my experience. Sorry others are still having issues. keithcurtis said: Ajax said: Working perfectly, I hardly think so, please don't piss down our backs and tell us it's raining... Please remain civil. The person you are quoting is a fellow user. I have pinged the team on this again.

I was initially agreeing with folks who were having difficulty after the fix - but I found my code had a silly error.  Things have been working pretty well for me since the fix.  Response time has slowed a little sometimes but overall much, much better than previously.  Sorry if this goes against others' experiences.  I must emphasize that I am just using API development and my player account, so I'm not seeing the fix under heavy usage. -- Tim

Yesterday evening the API was unresponsive only once, but is worked again after a restart. Guess i forgot to do the restart before playing - i suspect the issue is with a sandbox going idle/sleep/whateveryoucallit.

Yea, it starts to act unresponsive after a period of downtime.

I retract my earlier statement about things working perfectly. Today was a nightmare. After restarting the sandbox, I can get at most one api call before the sandbox hangs. Here's an example of a script hanging in the middle of execution. It's a detect magic script that draws and deletes animated rings around a token and pings sources of magic by filtering tokens looking for certain properties in GMnotes. You can watch the script (which used to work great) hang during the animation, requiring a manual deletion of the circles that exist when the sandbox hangs. No sandbox error messages. I can restart the sandbox and get the same behavior. Click to play gif:  

Earlier in the thread, Finds mentioned the API Heartbeat script. I've been using this as a workaround, I highly suggest grabbing it from one-click then activating it with !api-heartbeat. The consistent API usage has kept it from silently crashing in my game.

David M. said: I retract my earlier statement about things working perfectly. Today was a nightmare. Super pretty though! :)

Persephone said: Earlier in the thread, Finds mentioned the API Heartbeat script. I've been using this as a workaround, I highly suggest grabbing it from one-click then activating it with !api-heartbeat. The consistent API usage has kept it from silently crashing in my game. Thanks for this! Going to install it right now ahead of my session tonight.  And I also don't buy the Roll20 explanation about Chrome being the issue. I tested Firefox last night and ran into the same problem with big API lag.

Sent PM, thanks! Nicholas said: Hey folks -  We're currently investigating this, but need some additional information to better determine the root cause. Would you mind providing me the names or links of the games in which the API is having problems, please? You can send that to me in a PM if that is more convenient for you. Thanks in advance for your patience here!

Jay R. said: And I also don't buy the Roll20 explanation about Chrome being the issue. I tested Firefox last night and ran into the same problem with big API lag. Where did they say that this was a Chrome issue? I don't see it in this thread. I did see a report about Chrome recently changing some Canvas functions , which might affect LDL, are you thinking of that?

I'm having big problems with this, and my game just started.  It's my "D&D" game (yes, that's the game title).

keithcurtis said: Jay R. said: And I also don't buy the Roll20 explanation about Chrome being the issue. I tested Firefox last night and ran into the same problem with big API lag. Where did they say that this was a Chrome issue? I don't see it in this thread. I did see a report about Chrome recently changing some Canvas functions , which might affect LDL, are you thinking of that? Yep, I realized that I got two different issues mixed up. My bad. The Chrome thing pertains to general lag on LDL. My session tonight was BRUTALLY slow. Everything action or command took forever, even selecting tokens produced lag. And I disabled AFoW and DL on the map and it was still unacceptably slow. Please fix this, Roll20.

Once again terrible lag... I apologize to David M. as I thought he was a Roll20 Rep. I redirect that statement to whoever they trot out to tell us, "they're working on it"

API is totally dead for me at the moment. So.... no lag. Yey. :P

Our Thursday game was so slow we chatted while waiting on scripts to run. The script engine didn't crash, but it was that slow. I PMed the game link.

If this lag persists, I'm going to try a session using Firefox to see if Chrome is indeed the issue. The lag is distinct from the API crashing, which I managed to get around through Persephone's suggestion to install the APIheartbeat script. But both issues need to be fixed ASAP, Roll20. 

Almost unplayable. API stops if not used frequently. :( 

I have to wonder if the updated dynamic lighting has been worth all the trouble it's caused? Jay R. said: If this lag persists, I'm going to try a session using Firefox to see if Chrome is indeed the issue. The lag is distinct from the API crashing, which I managed to get around through Persephone's suggestion to install the APIheartbeat script. But both issues need to be fixed ASAP, Roll20.  I do my development in Firefox, we play a D&D campaign in Chrome.  We play using Chrome because Firefox does not work for any of us when it comes to being able to see each other and communicate. The lag and general API *&^%-up is very present in Firefox, Jay, I can assure you.  The campaign I'm working on has an important aspect that relies upon custom status markers to prevent players from moving infantry units marked as dug in or artillery units that are set to provide fire support.  While this works wonderfully when the game starts, a few minutes - I haven't figured out how long, the API dies.  :) Infantry battalions are able to frolic across the countryside, somehow dragging their trenches, bunkers, and other earthworks with them!  LOL My problem is that my friends will be plotting moves on their own and on different maps, so we won't have to be on line together.  There won't be a watchful GM to restart the sandbox when they are on plotting their units' moves. So again, Roll20, I really hope the dynamic lighting was worth it.  I'm going to see if there are other platforms out there I can use, shag this.  -- Tim

What does this current API glitch have to do with Dynamic Lighting?

keithcurtis said: What does this current API glitch have to do with Dynamic Lighting? Hi Keith.  I've spent more than 30 years supporting off the shelf database business/operations applications, in military, nuclear, and facilities maintenance environments.  I've seen new features sometimes breaking established functionality.  If the development team is not careful, further tweaking can wreck a lot of things. When they introduced dynamic lighting, the 3d dice from sendChat broke immediately.  That told me immediately that the dynamic lighting project made changes they did not fully investigate.  They've been tweaking dynamic lighting for some time.  My experience tells me that development direction and the the API problems are related. -- Tim

The key to the problem seems to be inactivity. If i restart a sandbox before play and keep the heartbeat API running... all goes well. It started after Roll20 had to scaleout the firebase infrastructure and the API sandbox infrastructure. So the API issues are probably a side effect of the scaleout of the API sandboxes and the firebase servers.  The API sandboxes run in nodejs servers and AFAIK do not contain the Dynamic Lighting code. These API scripts connect to the firebase servers and make changes there that are evented towards the clients.  As the logging of the API sandboxes is minimal for users, it is hard to debug as a user.  I suspect it is something like a connectionpool that returns NULL and then the error is discarded or leads to an endless wait (or something like that). Another thing might be that as the nr of firebase servers was increased from 60 to more than 200, it might be that connections to individual firebase servers now timeout and again the error is discarded or leads to an endless wait again. 

Hm, what I am experiencing is: restart the sandbox, and immediately call an api script. It either completes one script call and then hangs, or hangs the first time in the middle of the script (for the case of my script shown above that animates the expanding rings on a timed delay). No pink runtime error messages from the sandbox.

Tim M said: keithcurtis said: What does this current API glitch have to do with Dynamic Lighting? Hi Keith.  I've spent more than 30 years supporting off the shelf database business/operations applications, in military, nuclear, and facilities maintenance environments.  I've seen new features sometimes breaking established functionality.  If the development team is not careful, further tweaking can wreck a lot of things. When they introduced dynamic lighting, the 3d dice from sendChat broke immediately.  That told me immediately that the dynamic lighting project made changes they did not fully investigate.  They've been tweaking dynamic lighting for some time.  My experience tells me that development direction and the the API problems are related. -- Tim Good points, Tim. I certainly can verify that the API lag occurs on Firefox, but I am going to see if the general system lag -- separate from API, where even clicking tokens or non-API things in Chat takes an age -- can be mitigated by a change in browsers. But I do take your point. The work on UDL has had a negative impact on LDL already. Compounded by the fact that Roll20 seems to have laid off at least one in-house QA recently, which I find hard to understand given how many more users they've garnered during the pandemic (since playing F2F isn't an option for many people now).

Jay R. said: The work on UDL has had a negative impact on LDL already. Compounded by the fact that Roll20 seems to have laid off at least one in-house QA recently, which I find hard to understand given how many more users they've garnered during the pandemic (since playing F2F isn't an option for many people now). Software QA is what I was getting at, but didn't want to mention the term as my understanding is that the profession is being replaced - exactly by what, I don't know, but I'm not sure if my neighbour in Ottawa in the late 80s - who introduced me to the concept - is still doing it! :) All the best. -- Tim

David M. said: Hm, what I am experiencing is: restart the sandbox, and immediately call an api script. It either completes one script call and then hangs, or hangs the first time in the middle of the script (for the case of my script shown above that animates the expanding rings on a timed delay). No pink runtime error messages from the sandbox. Same exact here tonight, it is honestly the worst its been in weeks for us. I have to literally restart it every few minutes. Our game is impossible to play tonight. 3 hours in and we have made it about 10 minutes in game time.

Martijn S. said: The key to the problem seems to be inactivity. If i restart a sandbox before play and keep the heartbeat API running... all goes well. It started after Roll20 had to scaleout the firebase infrastructure and the API sandbox infrastructure. So the API issues are probably a side effect of the scaleout of the API sandboxes and the firebase servers.  The API sandboxes run in nodejs servers and AFAIK do not contain the Dynamic Lighting code. These API scripts connect to the firebase servers and make changes there that are evented towards the clients.  As the logging of the API sandboxes is minimal for users, it is hard to debug as a user.  I suspect it is something like a connectionpool that returns NULL and then the error is discarded or leads to an endless wait (or something like that). Another thing might be that as the nr of firebase servers was increased from 60 to more than 200, it might be that connections to individual firebase servers now timeout and again the error is discarded or leads to an endless wait again.  I really appreciate your taking the time to explain that, Martijn.  While my experience makes it impossible for me to ignore the conincidence, I'm probably old and set in my ways.  LOL You've certainly provided food for thought.  Not that it matters, I'm of no consequence here. I think I'm going to try the heartbeat script and see if that works before I give up completely.  FWIW, for those looking for it, like I was, it's APIHeartbeat, under Utilities in the script menu.  Let's see. :) Thanks again. -- Tim

The APIHeartbeat seems to help a little bit for us but not fix the issue. Now I only have to restart it every 10 minutes, instead of every time I need it. This is honestly the worst its been for us. The last month of games have been discouraging and frustrating. Everyone one of my players keep posting alternative options to Roll20 in discord chat at this point because they are ready to jump ship :(

We ended our session early, right in the middle of fight. There is just no way to play like this. 

DXWarlock said: The APIHeartbeat seems to help a little bit for us but not fix the issue. Now I only have to restart it every 10 minutes, instead of every time I need it. This is honestly the worst its been for us. The last month of games have been discouraging and frustrating. Everyone one of my players keep posting alternative options to Roll20 in discord chat at this point because they are ready to jump ship :( Yeah, the Heartbeat script helps, but it doesn't eliminate the problem completely. It's definitely better, in that I only have to restart the API every 10-15 minutes instead of every minute, but it's a band-aid on a gaping wound.

Sent you PM with the game link.  Let me know if there is anything else you need.  Thanks!

Game: D&D - 5e ( Original I know. )

Re-posted (my post) from a closed thread: Oddly, had no API lags or failures last night. Had general performance lags, but it was an enormous dungeon and the GM was using Advanced Fog of War. Everything  else  went wrong. One guy lost internet, two folks (including myself) had Discord utterly fail, and at one point I decided to restart my computer, and it crashed so hard that I had to run repairs in Safe Mode. Crazy night, but Roll20 chugged in through. I actually finished the night on my Chromebook running Discord through my phone. Weirdly enough, the Chromebook actually performed better than my Mac. Almost no lag. Go figure.

Well, I just did an experiment.  Firefox.  My roll20 campaign is not really an RPG, it's a table top wargame manager, so people go on at different times, do their moves on different pages and I adjudicate where contacts take place, sightings, etc.  Conflicts are resolved, when necessary, at my house on my wargames table with model tanks and figures, with indirect fire artillery support from artillery/mortar units in range of the battle.  So we are not on the campaign at the same time. Anyway, it's just in development now.  I had downloaded APIHeartBeat and enabled it and it seemed to keep the API going for me.  But after seeing some of the comments above, I turned on my Wargame Campaign moved a couple a couple of dug in units and artillery units set up to fire.  My APIs include a lot of on("change:graphic", function(obj, prev) { events that move units with specific token markers right back to where they started from.  All worked fine. I waited exactly an hour.  I came back, and the API was working fine. I then disabled the APIHeartBeat script.  Closed the browser.  Restarted and turned on the Wargame Campaign.  Initially, things worked as designed.  Ten minutes later, they did not. The only drawback about the very cool APIHeartBeat script, which won't apply to my Wargame Campaign, is that it continuously changes the players' colours.  The D&D game in which I'm about to take part (half an hour or so), that would be a but of a pain, as we can always tell who is rolling and who to congratulate or yell at for the result! :)  (Yes, it's on the chat window, but it's more fun to yell at the actual dice, LOL) -- Tim

I'm absolutely having API hang issues in my game:&nbsp; <a href="https://app.roll20.net/campaigns/details/8794747/by-the-will-of-the-icons" rel="nofollow">https://app.roll20.net/campaigns/details/8794747/by-the-will-of-the-icons</a> I'm literally only using one short script. It's taking ages to respond, when it does at all.

I tried to PM Nicholas, but I get "not authorized to post" when I do. :/ Here's my test game which is suffering from amazingly slow API.&nbsp;<a href="https://app.roll20.net/campaigns/scripts/7985900" rel="nofollow">https://app.roll20.net/campaigns/scripts/7985900</a>

@Nicholas - let's agree to disagree. It was working for a while, and just stopped. User error. Disregard.

Nicholas said: Hi everyone -&nbsp; The API servers should be back up and good to go. Please make sure to restart the API Sandbox in your game when you return to your adventures. Thank you! Thanks Nicolas - do you think the lag/timeout has been corrected?&nbsp;

Just did a test with APIHeartbeat disabled and I am cautiously optimistic. API seems to run fine. Did not experience any lag. But that's just a test run without any players in a real session. I'll see how it behaves next week.

API worked fine in my Sunday games, with the exception of ONE restart sandbox, which may have been at the same time Roll20 reset. Good report today, from my experience. There was a small lag / latency when I used !group-init --reroll The Group Initiative reroll command. It wasn't too bad. Maybe 10 seconds lag on that.

Sunday i added an every 30 second log statement to (my version of) api heartbeart script. Which has slower feedback than the alternating colors, but is good enough.&nbsp; And outside of the expected lag (1..10 seconds), everything worked without a glitch. Except of course players having crappy wifi, discord crashing three times, one player being late, another having a kid throwing up (and not muting his microphone). The normal stuff.

Nicholas said: Hi everyone -&nbsp; The API servers should be back up and good to go. Please make sure to restart the API Sandbox in your game when you return to your adventures. Thank you! Sweet! thanks! Hopefully that helps. Will see Saturday, as all week setting up the game was not TOO bad, but Saturday starting around 8pm EST is when it started coughing and gagging.&nbsp; Here is a link to our game if you are still interested in people submitting them: <a href="https://app.roll20.net/campaigns/details/3718260" rel="nofollow">https://app.roll20.net/campaigns/details/3718260</a>

The API isn't working at all today. Some of us have been posting in Pro and elsewhere. Please look into this.

It's working fine here.

I've been working on my game setup all morning and the API has actually been running well.&nbsp; I have no idea what the infrastructure is but maybe a certain shard is having issues?&nbsp;

I've had numerous instances of it lagging or freezing today without any crash.

<a href="http://status.roll20.net/" rel="nofollow">http://status.roll20.net/</a> Seems today has had a few glitches