API Inconsistently Responsive

The past week I've been having trouble with running any API commands. On a fresh start up the API seems to work fine, but every 20 minutes or so it just stops accepting commands until I restart it. I haven't had any crashes or error messages. And it also doesn't always spin up properly, sometimes I have to restart twice or more to get the console beyond the 'Spinning up new sandbox...' Sometimes the 'Restarting..' to 'Spinning up...' messages are in yellow text instead of the white and magenta it normally is. Not sure if that's linked to the issues I'm having, but I haven't noticed anyone report similar issues this week so I'm hoping for some help troubleshooting why my games are having this issue. Has anyone else been experiencing this?

I've been having hella lag on api commands today, too. Token-mod to set conditions taking around a minute. Similar where it works fine for a few minutes, then slows to a crawl. Sandbox restarts seem to take forever, too. Normal looking "spinning up" text for me, it just is taking a while. Luckily, I'm just playing around and not in the middle of a session. Hope this recovers soon.

Can confirm that it's behaving erratically. Sometimes it can take upwards of 8 minutes for the scripts to load up, where normally it would take about 20-30 seconds. I've just fired up my games about 30 minutes before scheduled start times and kept refreshing scripts with my fingers crossed. Can't do anything about them being non-responsive midway through a game though which is frustrating, as restarting in the middle of combat takes time for them to fire up again.

This happens in our games lately as well. Sometimes several times a session. I'll start directing reports here, and maybe we can shine some light on it. I would urge anyone with an unresponsive Sandbox that lacks a crash message to capture a console log for a  Help Center Request .

Yeah I can confirm it's happening to me as well today (Thanks to David M for letting me know I thought it was a script issue)

Happening to me as well.  I noticed it last weekend.   I have to constant restart the API server.   Today,  restarting the API server takes substantially longer than before.  No crashes today, just slow responses.

We've had this experience today as well, using the Star Wars FFG Dice Roller with the API. Not sure if it's the same issue of course, but we'll click "roll" and sometimes it goes through immediately and sometimes it takes 20 minutes. Luckily we have a discord bot to roll with as backups. It's happened in past sessions. Is there any way when running a game to get some sense of what's taking so long, a debug output or something similar?

Thanks for the suggestion, keith, I'll be sure to do that next time it happens. I'm surprised I hadn't seen it reported yet, but I'm glad to know I didn't mess up my console on my end somehow. Hopefully this can get resolved soon! :)

Last two weeks have been plagued by a slow to non-working API. 

API has been super-slow the past few weeks. Commands don't work without any error notice. Restarting takes an AGE and I'm not getting a console error about the API crashing. And I'm not wasting my time filing a help request. THIS THREAD should be enough.

I've experienced the same.  When writing APIs, compiling on a save is taking a long time most of this week.  Very often I just re-save while waiting for the API to compile - not sure if that's the correct term here.  Otherwise, "Spinning up new sandbox..." stays on forever .  I sometimes have to press the save button a few times. Look, I'm a new and a complete babe in the woods here, though I've been working in databases since 1988.  I don't know how stable things were in the past, but I'm thinking they may have accidentally busted up a few things with the new dynamic lighting; 3d Dice rolling via sendChat was an unfortunate early casualty. :(   Still love what they've given us, though. :)

API is BRUTAL tonight. Sandbox consistently crashes every 5 minutes with no error message. Big lag for any API use. Please fix, Roll20, ASAP.

I've had the API being unresponsive, requiring "Restart Sandbox" fairly often recently too. Concerned about this happening during Roll20CON when I'm planning to have really a lot of players in the tabletop, and would like to show off Roll20 with the cool API features, and 3D dice enabled, and webm animations on map Page, and Jukebox playing sounds, and FX particle effects. If any/all of those are slow, it could really grind a CON game to a halt.

This here too. I have restarted it at least 15 times in last hour as it just stops doing anything.

I've been having API lag and spinup issues for a couple weeks as well.  

iamk2 said: I've just fired up my games about 30 minutes before scheduled start times and kept refreshing scripts with my fingers crossed. Hi iamK2, what do you mean by refreshing scripts?  Is that the same as opening one up and clicking on save? -- Tim

Just wanted to add that I've seen this problem quit a bit lately as well. Restarting the sandbox seems to not complete for me about 40% of the time, requiring another restart. Crashes and code problems also don't seem to be producing error messages in the sandbox, so it just suddenly stops responding with no indication (other than that it isn't doing anything) that anything is wrong.

Tim M said: iamk2 said: I've just fired up my games about 30 minutes before scheduled start times and kept refreshing scripts with my fingers crossed. Hi iamK2, what do you mean by refreshing scripts?  Is that the same as opening one up and clicking on save? -- Tim They probably mean clicking the Restart API button on the scripts page. Common practice is to have a tab with that page up while you're in your Roll20 session, so you can quickly restart the API in case of crashes, which are happening a LOT lately.

Jay R. said: They probably mean clicking the Restart API button on the scripts page. Common practice is to have a tab with that page up while you're in your Roll20 session, so you can quickly restart the API in case of crashes, which are happening a LOT lately. Make sense to me!  Thanks. I don't know if it's the time of day, but it seems to be compiling relatively quickly for me at the moment - Sun Oct 18 2020 17:33:34 GMT+0000 (Coordinated Universal Time).

I haven't had too much issue with the API in my recent games...just a couple of "silent" API crashes which a "Restart API Sandbox" was able to fix...but today, it's just hanging: UPDATE: After a few minutes I reloaded the API page which kickstarted a proper API Sandbox restart and it seems to be working now.

I get a series of this: "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()" "Invalid character_id  for getAttrByName()"

I am having issues with unresponsive API as well. No error messages.

Same issue -- didn't understand why my tokenmod macros weren't working in my session yesterday. I checked today and I see what everyone else is seeing -- API sits at "spinning up new sandbox..." for quite a while.  Update: and now I also see what others see in the game -- even after spinning up the API, it just stops with no message or indication except that my token mod macros don't work. Only way to fix it is to restart the API. Macros work again, but for how long? The last session I ran where I saw no such problems was Sunday October 4. I didn't run any of my games until Sat Oct 17, which is when I first saw this issue.

I've had these same issues for awhile. What gives? 

Same thing here. API is really slow to spin up and suddenly stops. I'm constantly have to restart the API.

It does not happen to me as much as other people report, but in the last 5 hours i had to restart the sandbox once per hour or so.

I think that log captures would be of great help. At this point it's hard to tell whether this is a global API issue, or simply something wonky going on with a popular script. I don't have the forensic chops to tell the difference.

The problem is, the API Output Console looks something like: Restarting sandbox due to script changes... Previous shutdown complete, starting up... Spinning up new sandbox... "Starting webworker script..." "Loading 713 translation strings to worker..." "-=>HealthColors v1.6.1 [Updated: Aug 20 2020]<=-" "-=> TokenMod v0.8.61 <=-  [Tue Aug 11 2020 20:54:05 GMT+0000 (Coordinated Universal Time)]" "StatusInfo Ready! Command: !condition" <repeat from from first line> And quite often only the first three lines. For reporting purposes, is there script that pings every X seconds? Or logs every event (if we all install this, it will probably kill all API servers)

Not the API Output Console, the Browser Console: <a href="https://roll20.zendesk.com/hc/en-us/articles/360040673774-Submitting-a-Console-Log-in-Chrome#disclaimers-0-1" rel="nofollow">https://roll20.zendesk.com/hc/en-us/articles/360040673774-Submitting-a-Console-Log-in-Chrome#disclaimers-0-1</a> <a href="https://roll20.zendesk.com/hc/en-us/articles/360040673814-Submitting-a-Console-Log-in-Firefox" rel="nofollow">https://roll20.zendesk.com/hc/en-us/articles/360040673814-Submitting-a-Console-Log-in-Firefox</a>

Not sure if this will help with troubleshooting or not but I've noticed the freeze seems to happen during a lull of activity. When I'm constantly interacting with things that make use of the API I don't notice an issue until I stop for a bit and when I go back APIs have frozen. That being said I was using The Aaron's API Heartbeat script which is running a constant check and I haven't had a single freeze up for over 4 hours. I'd be interested to see if anyone else gets a similar result.

Confirming API is very slow over last couple days.

Well, here's some console information, at the bottom of my message.&nbsp; Using Firefox I opened up the campaign I am working. I then opened the API in a separate tab. I did not open any scripts in the API window. Clicked on Restart API Sandbox. Things seem to compile well.&nbsp; I only can judge success when the three Script LIbrary APIs I have: Spinning up new sandbox... "-=&gt; TokenMod v0.8.61 &lt;=- [Tue Aug 11 2020 20:54:05 GMT+0000 (Coordinated Universal Time)]" "-- True Page Copy v1.0 -- [Mon Apr 03 2017 18:32:50 GMT+0000 (Coordinated Universal Time)]" "-=&gt; WeaponArcs v0.1.3 &lt;=- [Tue Aug 04 2020 13:16:25 GMT+0000 (Coordinated Universal Time)]" I repeated 4 and step 5 occurred again very quickly. I decided to wait for a little while and watched TV/played around on Facebook for about 5 to ten minutes. I then c licked on Restart API Sandbox for the third time. A fair bit of time passed, at least ten minutes, before I got a similar message to 5, above.&nbsp; This was an unacceptable amount of time, of course. I then looked at the console.&nbsp; Nothing really seemed to have changed.&nbsp; I wish I had copied it beforehand to compare.&nbsp; Nevertheless,&nbsp; but between the two horizontal lines is what I copied.&nbsp; I hope this is of some help. This page uses the non standard property “zoom”. Consider using calc() in the relevant property values, or using “transform” along with “transform-origin: 0 0”. 8650008 Cookie “rack.session” has been rejected because it is already expired. 8650008 Some cookies are misusing the recommended “SameSite“ attribute 3 JQMIGRATE: Migrate is installed with logging active, version 3.0.0 v2.js:57:382 JQMIGRATE: jQuery.expr[":"] is now jQuery.expr.pseudos v2.js:57:129 console.trace() v2.js:57:178 &nbsp;&nbsp;&nbsp; i <a href="https://app.roll20.net/assets/v2.js?1599587595:57" rel="nofollow">https://app.roll20.net/assets/v2.js?1599587595:57</a> &nbsp;&nbsp;&nbsp; get <a href="https://app.roll20.net/assets/v2.js?1599587595:57" rel="nofollow">https://app.roll20.net/assets/v2.js?1599587595:57</a> &nbsp;&nbsp;&nbsp; &lt;anonymous&gt; <a href="https://app.roll20.net/assets/v2.js?1599587595:178" rel="nofollow">https://app.roll20.net/assets/v2.js?1599587595:178</a> &nbsp;&nbsp;&nbsp; &lt;anonymous&gt; <a href="https://app.roll20.net/assets/v2.js?1599587595:178" rel="nofollow">https://app.roll20.net/assets/v2.js?1599587595:178</a> &nbsp;&nbsp;&nbsp; &lt;anonymous&gt; <a href="https://app.roll20.net/assets/v2.js?1599587595:178" rel="nofollow">https://app.roll20.net/assets/v2.js?1599587595:178</a> Layout was forced before the page was fully loaded. If stylesheets are not yet loaded this may cause a flash of unstyled content. v2.js:51:25168 Logged in - checking Account GDPR status: true 8650008:384:11 Ignoring unsupported entryTypes: paint. 8650008:476:14586 No valid entryTypes; aborting registration. 8650008:476:14586 Ignoring unsupported entryTypes: largest-contentful-paint. 8650008:476:14620 No valid entryTypes; aborting registration. 8650008:476:14620 window.controllers/Controllers is deprecated. Do not use it for UA detection. ace.js:1:18446 JQMIGRATE: jQuery.fn.bind() is deprecated v2.js:57:129 console.trace() v2.js:57:178 &nbsp;&nbsp;&nbsp; i <a href="https://app.roll20.net/assets/v2.js?1599587595:57" rel="nofollow">https://app.roll20.net/assets/v2.js?1599587595:57</a> &nbsp;&nbsp;&nbsp; bind <a href="https://app.roll20.net/assets/v2.js?1599587595:57" rel="nofollow">https://app.roll20.net/assets/v2.js?1599587595:57</a> &nbsp;&nbsp;&nbsp; &lt;anonymous&gt; <a href="https://app.roll20.net/v2/js/browse.js?1599587595:609" rel="nofollow">https://app.roll20.net/v2/js/browse.js?1599587595:609</a> &nbsp;&nbsp;&nbsp; h <a href="https://app.roll20.net/assets/v2.js?1599587595:51" rel="nofollow">https://app.roll20.net/assets/v2.js?1599587595:51</a> &nbsp;&nbsp;&nbsp; d <a href="https://app.roll20.net/assets/v2.js?1599587595:51" rel="nofollow">https://app.roll20.net/assets/v2.js?1599587595:51</a> &nbsp;&nbsp;&nbsp; nrWrapper <a href="https://app.roll20.net/campaigns/scripts/8650008:476" rel="nofollow">https://app.roll20.net/campaigns/scripts/8650008:476</a> Automatically scrolling cursor into view after selection change this will be disabled in the next version set editor.$blockScrolling = Infinity to disable this message 14 ace.js:1:51429 Content Security Policy: The page’s settings blocked the loading of a resource at inline (“script-src”). 3 fingerprint-protection.js:270:52 Loading failed for the &lt;script&gt; with source “ <a href="https://js-agent.newrelic.com/nr-1167.min.js”" rel="nofollow">https://js-agent.newrelic.com/nr-1167.min.js”</a> . 8650008:1:1 Auth success! 8650008:2533:11 Automatically scrolling cursor into view after selection change this will be disabled in the next version set editor.$blockScrolling = Infinity to disable this message 16 ace.js:1:51429 I hope this is of some use. -- Tim

It's happened, repeatedly, three Sundays in a row for our gaming group.

Piss poor performance is getting to be the norm here, and nothing ever gets 100% fixed, we are paying for an inferior product, for crying out loud FIX THIS ISSUE it's been going on long enough!

For completeness will also report here (posted in another thread). Our group has been having worsening issues over the last few weeks, perhaps a month. The API will just stop responding. It's not crashed out, or errored it is just not answering. Last 2 weeks was by far the worst. On the 17th our Saturday game was more time spent with me kicking the API hoping to wake it up with resaves, or waiting for API chat responses when it was working (causing us to play the waiting game to see if it was dead, or just slow), than actual playtime. Restarting the API sandbox is a mixed bag of luck if it would actually finish a startup. -Sometimes it would start up fine and act as expected for up to 20 minutes. -Sometimes it was easy to tell as it would sit on "Spinning up new sandbox..." and nothing else, knowing I need to poke it again. -Sometimes it would spit out the obligatory script responses you expect in the log appearing its done and ready, but not actually respond to any chat calls. -Sometimes it would start up and respond, but with a long 5-15 second delay to commands.&nbsp;

Also suffering from this issue. Please get on this Roll20.

Thanks Nicholas!

Nicholas s Thank you, performance has greatly improved at my end. Noooowwwww, about the 3d dice from sendChat that Updated Dynamic Lighting broke? :) -- Tim

Been working perfectly since the update! THANK YOU!

Unfortunately I'm still experiencing slowness and 'silent' crashes.

Persephone said: Unfortunately I'm still experiencing slowness and 'silent' crashes. Same. Performance has improved a bit but the API still crashes silently and commands frequently take a long time to resolve.

Persephone said: Unfortunately I'm still experiencing slowness and 'silent' crashes. I can confirm that I'm still experiencing the same.

Persephone said: Unfortunately I'm still experiencing slowness and 'silent' crashes. Same here.

My api sandbox doesnt load in any game regardless of dynamic lighting settings.&nbsp; Is there somewhere that Roll20 announces downtime? I dont see anything on twitter or anything

Ajax said: Working perfectly, I hardly think so, please don't piss down our backs and tell us it's raining... Please remain civil. The person you are quoting is a fellow user. I have pinged the team on the API issue again.

Marionette said: &nbsp;Is there somewhere that Roll20 announces downtime? I dont see anything on twitter or anything Status reports: <a href="http://status.roll20.net/" rel="nofollow">http://status.roll20.net/</a> &nbsp;