Roll20 uses cookies to improve your experience on our site. Cookies enable you to enjoy certain features, social sharing functionality, and tailor message and display ads to your interests on our site and others. They also help us understand how our site is being used. By continuing to use our site, you consent to our use of cookies. Update your cookie preferences .
×
Create a free account

API Servers Tonight: Spinning Up Additional Capacity

We've had a pretty crazy day today so far...around 10k concurrents as soon as 4 PM (that's the most by far we've ever seen that early, and if that's any indication peak time tonight will be very busy). Apparently a lot of those are Pro users, as our API servers are running out of capacity. I am spinning up additional servers to handle the load, but the API may be unreliable for the next 15 - 30 minutes. I'll update this thread as I bring up the new servers. Thanks!
Additional capacity has been added and things should be running smoothly. Thanks!
1452387844
vÍnce
Pro
Sheet Author
Awesome!
1452389867

Edited 1452389906
DXWarlock
Sheet Author
API Scripter
Thats great news! Is it still ongoing? our scripts are intermittently working, and my "on ready" gm whisper keeps telling me the sandbox restarted every few minutes like I hit the save button.
Should be better now. Sorry for the restarts. 
1452393904
DXWarlock
Sheet Author
API Scripter
No problem at all! Thanks for all the work, the site seems LOT more snappy this saturday than its been in a while.
1452396127
DXWarlock
Sheet Author
API Scripter
Still getting sporadic API response, its switching between working, not working, and doubling up. Saving in the API window takes a minute or so to actually spin up, if it does at all.
Yeah it looks like one of the two API servers was still having some trouble keeping up last night. There was another spike around 9:20 PM (which is when you reported having issues DXWarlock) but then it looks like it stabilized a few minutes later. @Clinton R. Since you deleted your post I assume that means it is working for you now? All of the API Servers restart at 4 AM (which is when you posted) just to "purge" everything and make sure we're killing off any orphaned processes to start the new day off fresh. So there's a few minutes around that time when things may not work since hundreds of sandboxes are restarting at once, but that's normal. I'll continue to keep an eye on this today. We literally had about a 25% sudden increase in the amount of games being played at once yesterday, and while the new servers we put in place this week had no trouble keeping up with on-site activity, the API sandbox architecture was not ready for that sudden increase. I should note that we are in the process of completely rebuilding that architecture to use Docker containers instead of our current system, and that was what I was doing until all of this DDoS stuff required me to stop working on that and work on moving the servers around. We anticipate resuming work on that this week and we should have some more info (like a launch date) for that very soon. In the mean time I'll continue to apply temporary stop-gap measures on the existing sandbox architecture, but my guess is that last night was an anomaly and we won't see another sudden 25% jump like that for a while :-)
Better solution: Drop me an address of where this API feela is located am I will show up with a "Encouragement" stick.
1452457784
DXWarlock
Sheet Author
API Scripter
Ah, well that's a good thing lots and lots of users, more people playing tabletop :) A 'suddenly overload' situation for you, but in the end more people on Roll20 is a good thing. And thanks for the update, I wasn't complaining so much as just giving a heads up what I was seeing.
DXWarlock said: Ah, well that's a good thing lots and lots of users, more people playing tabletop :) A 'suddenly overload' situation for you, but in the end more people on Roll20 is a good thing. And thanks for the update, I wasn't complaining so much as just giving a heads up what I was seeing. Yep! Just wanted to let you know what was going on. The good news is we have had very nearly as many people on today as we did last night and all API Servers have been fine (*knocks on wood*). So like I said, hopefully we've at least got things at a point now where we can withstand the increased load until the new architecture is ready to roll out.
Riley D. said: @Clinton R. Since you deleted your post I assume that means it is working for you now? All of the API Servers restart at 4 AM (which is when you posted) just to "purge" everything and make sure we're killing off any orphaned processes to start the new day off fresh. So there's a few minutes around that time when things may not work since hundreds of sandboxes are restarting at once, but that's normal. Indeed. My problems were brief and then it was spinning up again. The reboot time makes sense, as I found it difficult to believe that API use could be overloading the servers at that hour of the morning. No harm, no foul. I remain a satisfied customer :)