Slow Responses

Hi, just wanted to check if anyone else is having a similar experience at the moment. I understand there are a lot more people using the servers at the moment, but the API slow down is fairly significant. Maybe 5 seconds longer on all responses, with every 10th maybe either taking 20-30 seconds or not completing at all.  I'm in Australia on the east coast using Roll20 between (5am - 10am) and I understand it's prime time in the US for gaming around then. Just wondering if I should be changing my schedule to accommodate the increased traffic, or if there's actually a known issue with the API? I have noticed this slowdown both yesterday (Sunday morning for me / Saturday night US), and again this morning (Monday AM / Sunday PM). Thanks. Byron.

OH yeah, last night we found it amusing after being frustrated at first. A couple of times it took 30 seconds to respond. I would imagine it is just volume.

Same here. API slowdown is comically bad at the moment.

I have definitely seen an intermittent lag between issuing a command and seeing a response. It comes and goes.

keithcurtis said: I have definitely seen an intermittent lag between issuing a command and seeing a response. It comes and goes. Yeah, the last 10 minutes have been great. The 10 minutes before that however, there were no responses at all. It's a minor problem for my game as a lot of the scripts running are modattr commands. So, if the delayed response seems like a lost command and we hit it again, and then it was just a lengthy delay, we sometimes get double modifications. Hardly a life threatening problem, I know. Which is why I'm happy to shift my timetable away from any particular time if I can understand it to be a traffic issue. Which it's sounding like it is. I'll just start recording approximate response times every couple of hours for a few days and see what comes of it. Thanks.

I noticed it a good deal last night during our game. It's frustrating, but I keep reminding myself that Roll20 is a web app running on a huge variety of browsers, platforms and connections, trying to do all kinds of things that browsers were not designed to specifically do. When stuff goes wrong, I switch into manual mode until it revives.

Last Saturday was a pain for us. I blame Covid-19.

Boz said: Hi, just wanted to check if anyone else is having a similar experience at the moment. I understand there are a lot more people using the servers at the moment, but the API slow down is fairly significant. Maybe 5 seconds longer on all responses, with every 10th maybe either taking 20-30 seconds or not completing at all.  I'm in Australia on the east coast using Roll20 between (5am - 10am) and I understand it's prime time in the US for gaming around then. Just wondering if I should be changing my schedule to accommodate the increased traffic, or if there's actually a known issue with the API? I have noticed this slowdown both yesterday (Sunday morning for me / Saturday night US), and again this morning (Monday AM / Sunday PM). Thanks. Byron. Experiencing the same thing, glad its not just me.

Dear Roll20, Buff the API please! Same experiences here, delayed macro results when API involved, worse at some periods of time than others.

Is this still a thing? I'm trying to decide if the delays I'm seeing in my game are related, or maybe because of one specific script or something. It's like the API just stops responding and then everything catches up several seconds later. It's not really noticeable with some stuff, but I've had to abandon several scripts I was using because the hiccups were too much of a distraction.

Same here, Hildy.  It's not all the time, sometimes the API scripts run and do what they're set to do, pretty quickly. Regular quickness. Sometimes it seems to lag and then blasts a few to screen suddenly in a row. To be honest I think this happens with 3D dice, and with Special FX, as well as with API scripts.  In the worst cases (most busy times?) even maps and regular dice rolls have been seen to lag. But clearly an ongoing issue with API scripts sometimes coming in delayed bursts.

Yeah, it likely happens more but we don't notice it. I use TokenCondition for light sources, though, so every time I hit a bump they'd stop moving with the players' tokens and then they'd all zip across the map and catch up. It was really obvious and felt like it was impacting player-confidence, so I abandoned the condition tokens and set light sources of the players tokens instead. It worked, I'm just sad that I can't use the bells and whistles that bring the game-space alive (for now).

i've had delayed blasts from Color-Emote API script, and group-initiative API script, two which sometimes we notice cuz waiting-expecting a rapid response. One of them rolls dice (group-init). Color-Emote doesn't even roll dice, but it does engage the Character, Controlled By: information.

Yeah, same. Real bad for the last 8 hours or so. I'm writing it off as traffic jams, and rescheduling things around it. From about now it'll be running like a dream - which I'm not writing off as a coincidence with Saturday Night gaming wrapping up. Every day, from about now it runs extremely well. It's a little circumstantial but I can't see it NOT being a factor.

For 2-3 weeks I've experienced slowdown in every game I've played, at some level.  Sometimes it's so bad we just give up on all macros/api and just go directly to the character sheet or /r.  Playing 4 times a week (weekends and weekdays both), and every game is noticeably worse than they were a month ago.  Not sure if the lag is due to the increase in Stay-at-home traffic or the release of the new lighting but me and my players are pretty disappointed.  Roll20, please put additional resources on this issue.  Thank you!

Scott P. said: For 2-3 weeks I've experienced slowdown in every game I've played, at some level.  Sometimes it's so bad we just give up on all macros/api and just go directly to the character sheet or /r.  Playing 4 times a week (weekends and weekdays both), and every game is noticeably worse than they were a month ago.  Not sure if the lag is due to the increase in Stay-at-home traffic or the release of the new lighting but me and my players are pretty disappointed.  Roll20, please put additional resources on this issue.  Thank you! Agreed. I'm running a game that is 100% reliant on the API as the combat scripts are running large equations with calculus - which I can't really ask the players to do on the fly - so it's essentially killed that campaign entirely. Which was a tough pill to swallow because it's taken 6 months to write it! Tough times though. Everyone's trying their best at most things so I'm not worrying too much about it at present and just trying different things to get around the slow periods.

Just wanted to add I've been experiencing the same, it's been a huge issue.

Definitely have been experiencing lag, with API and with other things. Created some confusion on Friday night when the combat tracker seemed unresponsive but then quickly moved through three players' turns. Have learned not to click it twice!

The API has been a pain today. Several seconds of lag before getting an answer.

Just an FYI for Turn Marker, it lags badly sometimes but restarting the sandbox always makes it immediately catch up and fix for a bit. Unfortunately restarting the sandbox every 5 minutes is utterly annoying.

Nicholas said: Hey folks -  We're still keeping an eye on this. We did have some connectivity problems last night which have since been addressed. Beyond that, was there any other times since I last checked in which you were experiencing these issues? Got This: API Output Console Your scripts are currently disabled due to an error that was detected. Please make appropriate changes to your scripts and click the "Save Script" button and we'll attempt to start running them again.  More info... For reference, the error message generated was:  TypeError: Cannot read property 'body' of undefined TypeError: Cannot read property 'body' of undefined at Request._callback (/home/node/d20-api-server/api.js:494:27) at self.callback (/home/node/d20-api-server/node_modules/request/request.js:185:22) at emitOne (events.js:96:13) at Request.emit (events.js:191:7) at Request.onRequestError (/home/node/d20-api-server/node_modules/request/request.js:877:8) at emitOne (events.js:96:13) at ClientRequest.emit (events.js:191:7) at TLSSocket.socketErrorListener (_http_client.js:358:9) at emitOne (events.js:96:13) at TLSSocket.emit (events.js:191:7) Does that help diagnose any of this?

More threads popping up every week asking about this. <a href="https://app.roll20.net/forum/post/8528807/api-scripts-running-very-slow-over-last-week" rel="nofollow">https://app.roll20.net/forum/post/8528807/api-scripts-running-very-slow-over-last-week</a> <a href="https://app.roll20.net/forum/post/8545386/very-slow-api-performance" rel="nofollow">https://app.roll20.net/forum/post/8545386/very-slow-api-performance</a> Nicolas / Roll20 Dev Team, please answer? Did the output console log tell anything about this issue?

Gold said: More threads popping up every week asking about this. <a href="https://app.roll20.net/forum/post/8528807/api-scripts-running-very-slow-over-last-week" rel="nofollow">https://app.roll20.net/forum/post/8528807/api-scripts-running-very-slow-over-last-week</a> <a href="https://app.roll20.net/forum/post/8545386/very-slow-api-performance" rel="nofollow">https://app.roll20.net/forum/post/8545386/very-slow-api-performance</a> Nicolas / Roll20 Dev Team, please answer? Did the output console log tell anything about this issue? Thank you for compiling these Gold! Roll20 Dev Team, please help!

My API has straight up stopped working and its now in permanent "previous shutdown complete" mode. No matter how long i wait or how often i restart it doesnt do anything.

I use TurnMarker in most of my games that have noticed API lagging. But, I was using TurnMarker for years on Roll20 before this recent spate, without seeing any API lag. Styx said: Just an FYI for Turn Marker, it lags badly sometimes but restarting the sandbox&nbsp;

I had the API crash at least 6 times tonight. I've got enough Roll20-fu to know to check, restart the sandbox and use other tricks to keep the action going. It must be frustrating for new users, though. Especially when this is not traditional behavior of the API.

API is so damn slow / crashy right now that it's really affecting gameplay. Roll20, you have to do something.

The most unusual thing was not the crashes themselves, but that about half of them gave no indication on the API page. No error message, just silent failure that required restarting the sandbox.

keithcurtis said: I had the API crash at least 6 times tonight. I've got enough Roll20-fu to know to check, restart the sandbox and use other tricks to keep the action going. It must be frustrating for new users, though. Especially when this is not traditional behavior of the API. Does Roll20 provide status of the API anywhere? I am getting error messages that indicate resource issues.&nbsp; runtime/cgo: pthread_create failed: Resource temporarily unavailable SIGABRT: abort I understand the load has increased significantly lately. I suspect they haven't ramped up their servers to meet the increased demand.

You mean the API in general? I don't think so. Each game&nbsp; has it's own sandbox, of course, but you could check for problems in any of the server shards here: <a href="http://status.roll20.net/" rel="nofollow">http://status.roll20.net/</a> It looks like there was a brief outage on several systems shortly after 9pm last night, GMT.

keithcurtis said: I had the API crash at least 6 times tonight. I've got enough Roll20-fu to know to check, restart the sandbox and use other tricks to keep the action going. It must be frustrating for new users, though. Especially when this is not traditional behavior of the API. This made me laugh for some reason because I'm exactly the same. I actually have my campaign API URL on my favourites bar and open it along with my campaign every time we play. I also made this extraordinarily simple power card for my macros bar:&nbsp; # !power {{ --name|API Message Check --whisper|self -- ?? @{GroupWOW|bard_level} &gt;= 1 ?? Get Request Confirmed| }} # Every time I don't get a response from the API I hit that. If it comes back instantly: gravy. If it doesn't - I flip to the API and tab and check. If that's cool - I just wait :)

Returned to roll20 after a hiatus, purchased a pro account and fired up the FFG dice roller and destiny scripts. Experienced 45 second delays on every 3rd or 4th roll. Really hurt the flow of a game and gave a whole bunch of new players a pretty poor intro to roll20.&nbsp; Is this just normal now? These are the only scripts I'm using so it's not clear to me if they are the issue?&nbsp;

Si said: Is this just normal now?&nbsp; Yea, it is normal. The API has been incredibly wonky for a month now and it makes it a pain to use scripts. Restarting your API usually does the trick.

Is there an open issue that roll20 are commenting on?&nbsp;

Si said: Is there an open issue that roll20 are commenting on?&nbsp; If you find one out, maybe we could organize a team to tell that link to the dozens of unanswered threads that have been asking about the same symptoms over the last few weeks under Bug Reports, Pro User Questions, and API board.

Hello everyone, We realize that you rely on Roll20 as a home for your games and we apologize for any interruption to your time playing on the site. We’ve been investigating Roll20’s sitewide response speed in several areas including the API. As we continue to improve our infrastructure with database and capacity changes, occasionally we uncover additional areas upstream that require work. We currently have a majority of the Roll20 development team devoted to resolving the problems you’re experiencing with the Virtual Tabletop response speed, the API Sandbox, long game load times, and slow dice rolls. If you have been having trouble with the API, please use this form to give us better insights into the problems you’ve encountered:&nbsp; <a href="https://forms.gle/FZZksovNtcfaBRi1A" rel="nofollow">https://forms.gle/FZZksovNtcfaBRi1A</a> This form will provide our team with the information we need to streamline, prioritize, and address API issues in the quickest and most effective way possible. Every report helps, so thank you for your patience as we continue to sort through this issue.

Thank you and the rest of the Dev team, Cavni. I know we don't always hear about how hard you folks are working, probably because you're too busy... working.

Hi Cavni, I have one question on how to answer the API insights form. Do you still want the report if we said YES, I restarted the API, and that fixed the problem . That is my question? In regards to the API Forum's question: Have you tried to restart the API Sandbox?&nbsp; * If you are unsure how to do this, please review the documentation seen here:&nbsp; <a href="https://roll20.zendesk.com/hc/en-us/articles/360037772753-API-Settings-Page-Overview" rel="nofollow">https://roll20.zendesk.com/hc/en-us/articles/360037772753-API-Settings-Page-Overview</a> Yes No