This post has been closed. You can still view previous posts, but you can't post any new replies.

API Issues (Refreshing/Down) Saturday 1/31

1422733812

Permalink Quote

As the title suggests, the API server keeps spinning up a new sandbox when nothing is going on. Additionally, some scripts work perfectly well for awhile, then randomly crash the sandbox, forcing me to disable the script. After awhile, I can re-enable the script, and everything works fine with the script and everything again, then it starts crashing the sandbox again. Throughout the entire time, the sandbox continually spins up a new one, every few minutes.

1422735629

Permalink Quote

It just crashed now, for instance, and gave me this error: /home/symbly/www/d20-api-server/node_modules/firebase/lib/firebase-node.js:1 orts, require, module, __filename, __dirname) { function f(a){throw a;}var j=v ^ Error: Firebase.child failed: First argument must be a non-empty string and can't contain ".", "#", "$", "[", or "]". at Error (<anonymous>) at Ha (/home/symbly/www/d20-api-server/node_modules/firebase/lib/firebase-node.js:12:204) at G.W.H (/home/symbly/www/d20-api-server/node_modules/firebase/lib/firebase-node.js:126:213) at TrackedObj._doSave (

1422736290

Pro

Permalink Quote

That error would seem to indicate a problem with your script...

1422736589

Edited 1422736649

Permalink Quote

I would agree, but the script has been disabled for a few hours. It just keeps happening randomly. I disabled the script awhile ago because I got tired of it working for an hour, then suddenly not working, causing the server to crash.

1422738985

Edited 1422739113

Sheet Author

API Scripter

Permalink Quote

Sean G. said: Throughout the entire time, the sandbox continually spins up a new one, every few minutes. Not getting any errors, but came to see if I was the only one with the sandbox randomly restarting and saw this thread. I have my on:ready /w me when sandbox starts. I've been idling in the game waiting for tonight, I've gotten about 10-15 /w messages that the sandbox has started. For me its not effecting anything or any adverse effects, just strange. For example Ive gotten 2 while reading this thread and posting I can hear the chat beep in the background when it whispers me. I was assuming Riley or someone was working on the dev server API and restarting them or such. Or its some part of the watchdog (if there is one now) that kills duplicate sandboxes at work.

1422739249

Permalink Quote

Nah, it's constantly happening. It's killing API scripts during game play right now. PowerCards keeps double kicking out, or not at all, turntracking APIs stop functioning.

1422739707

Permalink Quote

My scripts are not working anymore either. Nothing changed on my end...

1422740086

Sheet Author

API Scripter

Permalink Quote

I spoke too soon, mine arent working now at all either.

1422740168

Permalink Quote

Yeah, we've been having issues where it stops working for about 15 minutes, while the API server spins up 15 times or so, then the words come across my screen saying the API server restarts, and scripts start working again for a little while.

1422740496

Edited 1422740527

Sheet Author

API Scripter

Permalink Quote

Looks like the same thing I got now. API will respond fine, Then, I get random 'Spinning up new sandbox...' for no reason every couple minutes, then its nonresponsive. Eventually it will respond to calls again.

1422745281

Edited 1422745683

Roll20 Team

Permalink Quote

I just gave the server a kick, hopefully that will help. Honestly, it looks like it's just overloaded right now. Our primary server process (the one that dispatches all the sandboxes) is pegging very close to 100% CPU, so it's very possible that we're just reaching the maximum number of API sandboxes we can handle. I will be spinning up additional servers to help with this during our planned maintenance on Monday (see: <a href="http://blog.roll20.net/post/109611193945/scheduled-maintenance" rel="nofollow">http://blog.roll20.net/post/109611193945/scheduled-maintenance</a> ), but unfortunately there's not a whole lot I can do right at this moment...we planned to increase capacity but we may be doing it just a few days late. I will continue keeping an eye on it and do what I can, though! EDIT: I just looked and things seem much better since the restart. We do have 125 sandboxes running currently which is the most we've ever had running at once, so there may be growing pains (i.e. we're in uncharted waters here). As I said I'll continue to monitor throughout the evening as we hit peak time.

1422749273

Permalink Quote

I love the kick. :)

1422749325

Permalink Quote

It does seem to be working a little bit better now, since you gave the kick.

1422750803

Sheet Author

API Scripter

Permalink Quote

Thanks Riley, it seems to be running ok now for me also.

1422754901

Roll20 Team

Permalink Quote

We're still tracking this issue as I said. Sorry that the post got closed. Please feel free to post in there throughout the night and let me know if you have further issues. I just kicked it again.

1422755022

Edited 1422755030

Sheet Author

API Scripter

Permalink Quote

The kick didn't seem to help mine, it went from double posting, to not replying at all now.

1422755051

Roll20 Team

Permalink Quote

William R. said: The kick didn't seem to help mine, it went from double posting, to not replying at all now. It seems to be in a crashing loop. I'm going to take it down for now and try to put some temporary fixes in place. Stand by.

1422755110

Permalink Quote

Thank you sir. Thank you for letting us know.

1422755265

Sheet Author

API Scripter

Permalink Quote

Yes thank you We really aren't trying to bring the site to its knees, we promise :)

1422755656

Roll20 Team

Permalink Quote

I just pushed out a fix that seems to have fixed the crashing loop. I'll keep monitoring.

1422755728

Permalink Quote

I currently am seeing 3 powercards per action since the fix.

1422755775

Edited 1422755899

Sheet Author

API Scripter

Permalink Quote

Kevin said: I currently am seeing 3 powercards per action since the fix. Try hitting save, I was getting 2. A save fixed mine.

1422755977

Permalink Quote

We logged for the night from our campaign, so I won't be on checking up on it. 12 hours of play, with a lot of API breaks wears you down :P. Anyhow, when we logged, it had reset, and things were rolling, just not sure if it was constant or not. I did get one unexpected token error before logging, however, which (again) is odd because none of the scripts had changed, or been enabled/disabled during the entire day (save one that was giving random errors, and random times that was disabled about 6.5 or so hours ago).

1422756080

Roll20 Team

Permalink Quote

Sean G. said: We logged for the night from our campaign, so I won't be on checking up on it. 12 hours of play, with a lot of API breaks wears you down :P. Anyhow, when we logged, it had reset, and things were rolling, just not sure if it was constant or not. I did get one unexpected token error before logging, however, which (again) is odd because none of the scripts had changed, or been enabled/disabled during the entire day (save one that was giving random errors, and random times that was disabled about 6.5 or so hours ago). Sorry about that! As I said we'll be definitely making some upgrades Monday. Just wish we would have gotten to it a bit sooner apparently.

1422756108

Permalink Quote

I attempted a save then got the following: Spinning up new sandbox... Unexpected token s Unexpected token w It seems now that after 2-3 saves, it seems to have normalized now.

1422756193

Roll20 Team

Permalink Quote

William R. said: Kevin said: I currently am seeing 3 powercards per action since the fix. Try hitting save, I was getting 2. A save fixed mine. Indeed, if you see any lingering issues a Save should clear them up.

1422756684

Permalink Quote

Hasn't worked in the game I am in - watching the API page, the DM has told us it repeatedly is "restarting the sandbox" on it's own. Saving in my game I DM last night, I had the same situation for about 20 minutes. No API functions. Come on MONDAY!

1422757422

Edited 1422757450

Permalink Quote

Riley, don't apologize. We left iTableTop half a year ago because there were no updates, no chance to make any sort of modifications for personalization, and no support... You guys are doing a hell of a damn good job on this! It was just really bad timing since we only play every other Saturday. So far, this has been the one day, of all the days we have played, where there has been problems like this. So I certainly won't sweat it, and I know my players won't either (of course, I have an advantage in that department since they are all family members except one). Seriously, you guys should be very proud of where you are at. Needing to upgrade and add new servers because the current ones are being overloaded due to too many people, is something few places can make claim on.

1422757500

Permalink Quote

And Kevin, yeah -- those were the unexpected token errors I was getting also, except I had one that also was a ;. It was a little bizarre.

1422757910

Edited 1422757939

Sheet Author

API Scripter

Permalink Quote

Sean G. said: Riley, don't apologize. We left iTableTop half a year ago because there were no updates, no chance to make any sort of modifications for personalization, and no support... You guys are doing a hell of a damn good job on this! It was just really bad timing since we only play every other Saturday. So far, this has been the one day, of all the days we have played, where there has been problems like this. So I certainly won't sweat it, and I know my players won't either (of course, I have an advantage in that department since they are all family members except one). Seriously, you guys should be very proud of where you are at. Needing to upgrade and add new servers because the current ones are being overloaded due to too many people, is something few places can make claim on. My thoughts exactly! the small issues we run into isnt even anything for us to stress over vs what we get out of you guys making the site.

1422757911

Roll20 Team

Permalink Quote

OldSchoolChris said: Hasn't worked in the game I am in - watching the API page, the DM has told us it repeatedly is "restarting the sandbox" on it's own. Saving in my game I DM last night, I had the same situation for about 20 minutes. No API functions. Come on MONDAY! What campaign is it? Maybe I can take a look if you're still wanting to play tonight...

1422759629

Edited 1422759722

Sheet Author

API Scripter

Permalink Quote

An update, on normal /roll rolls sometimes we are getting: There was an error fetching your roll. Please try again.undefined These aren't API rolls just the built in [[1d6]] or /roll 1d6

1422761216

Roll20 Team

Permalink Quote

William R. said: An update, on normal /roll rolls sometimes we are getting: There was an error fetching your roll. Please try again.undefined These aren't API rolls just the built in [[1d6]] or /roll 1d6 Yeah there was about a 2 min hiccup with the QuantumRoll engine for 3D Rolls on Dev. Should be fixed.

1422801536

Pro

Permalink Quote

Last night no scripts were working consistently, DoorControl-zip, BlindRoll-worked intermittently, BloodAnd Honor-zip, in the past when they crashed the API crashed, last night it was too bust "starting new sandbox" to do anything else. All of these scripts have been working in the past. This would be the game OldSchoolChris mentioned in his post.

1422814993

Roll20 Team

Permalink Quote

After the emergency fix we pushed out last night things seemed to go much better. We'll keep an eye on it today but please feel free to make a new post if you run into additional issues. Thanks!

1

DriveThru

DMs Guild

Roll20 Characters

Dungeon Scrawl

Pathfinder Infinite

Starfinder Infinite

Roll20^® is a Registered Trademark of Roll20, LLC. All rights reserved.