Help With Mysterious, Periodic Error That Appears Randomly?

For months now, I'll often return to my game to find the API has crashed with this error: API Output Console Your scripts are currently disabled due to an error that was detected. Please make appropriate changes to your script's code and click the "Save Script" button. We will then attempt to start running the scripts again.  More info...  If this script was installed from the Script Library, you might find help in the Community API Forum. For reference, the error message generated was:  TypeError: Cannot read property 'body' of undefined TypeError: Cannot read property 'body' of undefined at Request._callback (/home/node/d20-api-server/api.js:500:28) at self.callback (/home/node/d20-api-server/node_modules/request/request.js:185:22) at Request.emit (events.js:310:20) at Request.onRequestError (/home/node/d20-api-server/node_modules/request/request.js:877:8) at ClientRequest.emit (events.js:310:20) at TLSSocket.socketOnEnd (_http_client.js:453:9) at TLSSocket.emit (events.js:322:22) at endReadableNT (_stream_readable.js:1187:12) at processTicksAndRejections (internal/process/task_queues.js:84:21) I haven't been able to find anything in the console or my (extensive) debug logs that provide any messages, alerts or other indications of why this might be happening, and nothing in the stack trace references anything in my scripts and I don't understand what it's saying so I can't really figure out how to debug it. Is anyone able to help clarify what this error might be trying to tell me?

Is anyone able to help clarify what this error might be trying to tell me? Start by listing all the APIs you have installed in the game, and people can start giving suggestions. For months now, I'll often return to my game to find the API has crashed with this error: are there any more problems, apart from the crashed API sandbox? are you saying this happens when you aren't actively using the game? how often have this occurred? Have you tried turning off some of the APIs one-by-one to try narrow down which API might be the source of the error-message?

If you can't find anything in the api scripts, it might be coming from your character sheet. Character sheet Sheet Workers are also executed on the API sandbox. What scripts and Character sheet are you using?

Aaron, can sheet workers errors crash the API sandbox?

GiGs said: Aaron, can sheet workers errors crash the API sandbox? Yep, just try using a nullish operator or optional chaining in a character sheet. You'll get a syntaxError in your API sandbox that will crash it.

Oh, how I wants the nullish operator, precious... ...gollum... ... gollum

Scott C. said: GiGs said: Aaron, can sheet workers errors crash the API sandbox? Yep, just try using a nullish operator or optional chaining in a character sheet. You'll get a syntaxError in your API sandbox that will crash it. Thanks. I have seen errors from the character sheet show up in the API sandbox but have never managed the API sandbox. I need to up my game!

Sorry for the late reply all!&nbsp; Thanks a ton for your comments! Andreas J. said: Start by listing all the APIs you have installed in the game, and people can start giving suggestions. Yeah, about that... I've been running this game for nearly three years and have coded everything myself.&nbsp; Right now I'm up to 39,305 lines of custom code across 21 scripts.&nbsp; The only script I've borrowed from the community is AirBag, and even then I've modified that significantly.&nbsp; (Edit:&nbsp; I do have a GitHub repo that includes all of my scripts if you'd like to take a look ---&nbsp; <a href="https://github.com/Eunomiac/Vampire5E" rel="nofollow">https://github.com/Eunomiac/Vampire5E</a> &nbsp;--- but I hardly expect anyone to scour through them for the solution!) Andreas J. said: are there any more problems, apart from the crashed API sandbox? Nothing I haven't been able to explain/understand/fix --- all that code seems to work perfectly aside from this. Andreas J. said: are you saying this happens when you aren't actively using the game Yes: I'll return to the game after several days and find this error waiting for me.&nbsp; It occasionally occurs during play, but I've modified AirBag so that it automatically handles non-fatal errors without shutting down the API, so I typically don't notice (as this error doesn't appear to prevent the API from functioning on its own).&nbsp; However, when no one is logged in, AirBag doesn't activate to catch the error and the API shuts down.&nbsp; This is bothersome because, among other things, I have a countdown on the main splash page that counts down to the next session and the scripts need to be on (or able to spin up) whenever someone logs on. Andreas J. said: how often have this occurred? Mmm... because it's so inconsistent, and because it reveals itself only when I log in to check (and doesn't timestamp when it happened) it's really hard to say.&nbsp; I'd estimate weekly?&nbsp; Ish?&nbsp; But for all I know it could fire daily and I just haven't noticed since I'm not always logged in and watching. Andreas J. said: Have you tried turning off some of the APIs one-by-one to try narrow down which API might be the source of the error-message? This particular error is annoyingly resistant to that kind of debugging, since I have very little control or understanding over when it appears.&nbsp; I suppose I could leave each script off for a week or two at a time to see whether one or the other causes the issue, but with so many scripts that'd take months. The Aaron said: If you can't find anything in the api scripts, it might be coming from your character sheet. Character sheet Sheet Workers are also executed on the API sandbox. What scripts and Character sheet are you using? One I built myself (for Vampire 5th edition), which includes 4000 lines of sheetworker javascript. The reference to "body" in the error makes me think it might be something to do with the character sheet, but I haven't made any significant changes to the sheet in a long while... of course, this error has been vexing me for a long while too, so I can't be 100% sure. I realize I'm not being the most helpful patient for your tender debugging ministrations, but I hope I've shown why I'm having such trouble pinning this down!&nbsp; I was really hoping someone with knowledge of the Roll20 inner workings might be able to glean something helpful from the stack trace or error report itself?

Here is a link to my github repo where you can find all of my scripts and my character sheet, if for god knows what reason you're so inclined ;) <a href="https://github.com/Eunomiac/Vampire5E" rel="nofollow">https://github.com/Eunomiac/Vampire5E</a> ... and here's the order of my scripts:

Eunomiac said: ... and here's the order of my scripts: FYI, the order of the scripts in your Script panel is not guaranteed to match their installation/run order. I've had it in a few games I've had to troubleshoot that messing with a script (installing, enabling, disabling) causes the scripts around it to reorder. The best way I've found to know what order the scripts will run is by using Aaron's API_Meta trick. Put this at the top of your script, preferably below a comment line (to protect against poorly terminated previous script lines): var API_Meta = API_Meta || {}; API_Meta.&lt;scriptname&gt; = { offset: Number.MAX_SAFE_INTEGER, lineCount: -1 }; { &nbsp; &nbsp; try { throw new Error(''); } catch (e) { API_Meta.&lt;scriptname&gt;.offset = (parseInt(e.stack.split(/\n/)[1].replace(/^.*:(\d+):.*$/, '$1'), 10) - (13)); } } Replace &lt;scriptname&gt; with the unique name of this script, and replace the 13 at the end of the try/catch line with the line number where that line occurs (ie, for me, this line would have appeared on line 13). Then at the end of your script, you can put this: { try { throw new Error(''); } catch (e) { API_Meta.&lt;scriptname&gt;.lineCount = (parseInt(e.stack.split(/\n/)[1].replace(/^.*:(\d+):.*$/, '$1'), 10) - API_Meta.&lt;scriptname&gt;.offset); } } Again, replacing &lt;scriptname&gt; . Once you have that, if you put a log/registration line in your script, you can output that offset to know where a script begins in the concatenated file of all your loaded &amp; enabled scripts: Which is also super-helpful for debugging, but it can tell you for sure what order your scripts are installed in.

timmaugh said: Eunomiac said: ... and here's the order of my scripts: FYI, the order of the scripts in your Script panel is not guaranteed to match their installation/run order. I've had it in a few games I've had to troubleshoot that messing with a script (installing, enabling, disabling) causes the scripts around it to reorder.&nbsp; Oh, I definitely don't rely on the order of my script tabs to handle loading and initialization --- see that "InitCommands.js" script?&nbsp; It's whole purpose is to "conduct" the initialization of the API as a whole, ensuring any scripts that need to be loaded first get preinitialized or whatnot.&nbsp; (There's only one 'on("ready")' call anywhere in my scripts.) timmaugh said: The best way I've found to know what order the scripts will run is by using Aaron's API_Meta trick. Have you taken a look at AirBag.js?&nbsp; It uses two scripts, one you put as your first script (on your API tab), and the second you put as your last script (in this case, the order does matter, as the first script starts an unfinished try/catch block that wraps every other script you load, until the second AirBag script at the end closes it).&nbsp; Then, you just need to put MarkStart(" &lt;scriptName&gt; ") at the top of each script, and&nbsp; MarkStop(" &lt;scriptName&gt; ") at the bottom. AirBag also logs line numbers, order of initialization and any registered event listeners at API start-up.&nbsp; But its stand-out features for me are: The vast majority of errors do not cause the API to shut down --- they're caught by AirBag's try/catch block, reported to the game GM via chat message, and then a button on the error message allows the GM to restart the API from within the game. You can also edit the script to either restart automatically on an error, or even ignore errors entirely and keep running as if nothing happened (I've had surprising success with the last part --- once caught, few errors are enough to bork up AirBag itself) The stack trace report for any thrown error converts all line numbers to the local line numbers inside each of your scripts: ... though only in the fork I made of VoltCruelerz' original , which hasn't been updated in a few years.&nbsp; You can check out my fork here:&nbsp; <a href="https://gist.github.com/Eunomiac/6cab31a2eacbf99d3fc2edfc40272e1d" rel="nofollow">https://gist.github.com/Eunomiac/6cab31a2eacbf99d3fc2edfc40272e1d</a>

Since we're talking mysterious periodic errors... This seems to pop up nearly every time my players go into initiative. It's likely that one of them is hitting initiative without a selected token, but that shouldn't crash the API. I suspect you guys will look at this and say "No way of knowing" buuuuut... just in case... I'm running:

If I had to guess, I'd say that one of the scripts doesn't support Custom Turns in the Turn Order.&nbsp; My guess is that whatever MarkTurnStartLocation.js is, it begins by grabbing the token that represents the current turn.&nbsp; In the case where the top turn is a custom turn, that wouldn't be a token and would return undefined, then there might be a reference to the id that fails.&nbsp; Hard to know without looking at that script.&nbsp; See if it happens right when you cycle the turn with a custom turn entry coming to the top or going to the bottom or what have you.&nbsp; If that's the case, try disabling half your scripts and see if it still happens.&nbsp; Keep doing that until you figure out which script is the culprit.

Hmmm... typically after a fight, I'll clear the initiative out completely, so when we head into initiative (and this error gets generated), there are no custom turns in place at all. I think your method might be the only one that works... although it's also easy enough to just restart the sandbox. A bit annoying, but nothing tragic.

i basically just restart the sandbox before every session.&nbsp; unfortunately, i've found that "API" codes are generally brittle and cause frequent crash.&nbsp; it's partly also the messy way in which roll20 implements their "API" system.&nbsp;&nbsp;

aisforanagrams said: i basically just restart the sandbox before every session.&nbsp; unfortunately, i've found that "API" codes are generally brittle and cause frequent crash.&nbsp; it's partly also the messy way in which roll20 implements their "API" system.&nbsp;&nbsp; This just isnt true. Some APIs have bugs in them, which havent been discovered and fixed yet, which can lead to crashing. But you can go forever without encountered a sandbox crash if you use only reliable and well-tested API's. The problem is that the roll20 environment makes it very hard to identify which of your scripts are causing crashes, and everyone's environment is unique - which means if you're running a lot of scripts, you have to do some error testing yourself to find which script is bugging out, and that can be hard work and most people don't know how to do it.

GiGs said: aisforanagrams said: i basically just restart the sandbox before every session.&nbsp; unfortunately, i've found that "API" codes are generally brittle and cause frequent crash.&nbsp; it's partly also the messy way in which roll20 implements their "API" system.&nbsp;&nbsp; The problem is that the roll20 environment makes it very hard to identify which of your scripts are causing crashes, and everyone's environment is unique - which means if you're running a lot of scripts, you have to do some error testing yourself to find which script is bugging out, and that can be hard work and most people don't know how to do it. This is pretty much the boat I'm in.&nbsp; Usually I can debug scripts myself with some effort --- disabling scripts, commenting out large chunks, log messages, basically all the standard tricks.&nbsp; Do you have any advice for me in this particular instance, where the combination of (A) an unhelpful error message, (B) my inability to reliably reproduce the error, and (C) the fact that would take several days to test the effect any changes (i.e. disabling scripts/functions) might have, means I can't use any of the methods I typically rely on? Is this a matter for Roll20 Support, do you think?

For my own scripts, I have a mock version of the Roll20 API that I can instrument and set the internal state of while I'm working on scripts.&nbsp; I use Jest as a testing framework and extract portions of scripts to subject them to a battery of specific tests to make sure they do what I want them to do.&nbsp; I spend a lot of time thinking about the design of my scripts (sometimes to the point that I don't end up finishing them...), and how to decompose them into understandable and testable parts.&nbsp; All of that feeds into making scripts that I don't have any trouble reasoning about, and where I have generally caught all the major problems either by virtue of having designed them sufficiently, or by having tested them enough. When it comes to other peoples' scripts, finding problems can be quite a bit more difficult.&nbsp; When you have an (A) unhelpful error message, there's not much you can do.&nbsp; You can usually divide it into at least 2 possible categories: 1) In an API script somewhere or 2) In the Roll20 side.&nbsp; If it's on the Roll20 side, you can sometimes figure out how it got there (NaN in property usually means a problem in some set you did in an API script, for example), but sometime you're just out of luck.&nbsp; If you can tell that it's in an API script, sometime you can look for a unique sequence of known functions (map, followed by reduce, followed by foreach, followed by set...) and look at which scripts have those functions in that order.&nbsp; When reproducing an error (B) , you really have to get in the habit of watching your API console in a separate window while you're gaming, and keep track of what you were doing right when you get the crash message.&nbsp; I keep the API console up while working on a script, and also while running a game.&nbsp; If I get a crash, I try to reason about what I was just doing, and come up with experiments for what I think might have caused an issue.&nbsp; I'll write little repro scripts to try and show the problem, which can then be used to either fix the problem in the script, or send to Roll20 if the issue is in their code.&nbsp; With the time factor (C) , I don't really have much in the way of suggestions.&nbsp; Functional Decomposition in your own scripts, and extra testing can make it faster to find issues.&nbsp; Clean Coding habits can help avoid some common issues, or cause error messages to be more useful.&nbsp; Having a good understanding of what events each script is listening to, or what systems it is affecting can help you narrow down where a problem might be faster. I hope that helps.&nbsp;&nbsp;

GiGs said: aisforanagrams said: i basically just restart the sandbox before every session.&nbsp; unfortunately, i've found that "API" codes are generally brittle and cause frequent crash.&nbsp; it's partly also the messy way in which roll20 implements their "API" system.&nbsp;&nbsp; This just isnt true. Some APIs have bugs in them, which havent been discovered and fixed yet, which can lead to crashing. But you can go forever without encountered a sandbox crash if you use only reliable and well-tested API's. The problem is that the roll20 environment makes it very hard to identify which of your scripts are causing crashes, and everyone's environment is unique - which means if you're running a lot of scripts, you have to do some error testing yourself to find which script is bugging out, and that can be hard work and most people don't know how to do it. Well, all i know is that in my experience with roll20's api system, every game where i use even couple of API codes crash the sandbox with relative frequency.&nbsp; And honestly restarting is easy enough that the effort to debug is much less appealing to me.&nbsp; The point is, there's little guarantee that any QA effort goes into scripts that are accepted to their library and I'm honestly not about to do it for all the scripts I use.

Eunomiac said: GiGs said: aisforanagrams said: i basically just restart the sandbox before every session.&nbsp; unfortunately, i've found that "API" codes are generally brittle and cause frequent crash.&nbsp; it's partly also the messy way in which roll20 implements their "API" system.&nbsp;&nbsp; The problem is that the roll20 environment makes it very hard to identify which of your scripts are causing crashes, and everyone's environment is unique - which means if you're running a lot of scripts, you have to do some error testing yourself to find which script is bugging out, and that can be hard work and most people don't know how to do it. This is pretty much the boat I'm in.&nbsp; Usually I can debug scripts myself with some effort --- disabling scripts, commenting out large chunks, log messages, basically all the standard tricks.&nbsp; Do you have any advice for me in this particular instance, where the combination of (A) an unhelpful error message, (B) my inability to reliably reproduce the error, and (C) the fact that would take several days to test the effect any changes (i.e. disabling scripts/functions) might have, means I can't use any of the methods I typically rely on? Is this a matter for Roll20 Support, do you think? I would normally say that Roll20 Support wont be able to help you because its not actually a problem with roll20's basic features, it's the content you have created in roll20. They can't debug your code for you. That said, the error message you posted at the start seems to point to roll20 code, nit yours, if I'm reading it right, so maybe contacting them might help. I'm not sure how much help I can personally suggest - a quick glance at your code suggests you have decent code skills (likely better than mine!). So you're at the stage of just doing the hard work - trying to break you code down into small manageable chunks, and testing until you find what breaks. One thing you can do as well as logging, is add whispered sendChat status reports- maybe go through and add a lot of these, at points where processes are happening. It'll add a lot of spam to your chat log, but when the code crashes, they'll stop being sent. So you might be able to trace the process that led up to the crash. Knowing when they've stopped might help you figure out what is causing the error. With a codebase as big as this, there is no easy solution. With intermittent errors, you need to do some serious debugging, and the amount of your script code makes that a gargantuan job. One thing that puzzles me is that you are coming back and finding the API disabled. Does this only happen when you are out of a session, or does it happen during sessions too? If the latter, my guess is it's happening when someone else logs into the campaign and does something with their characters. Practical suggestion: I'd try to sort out which scripts I can do without, and disable them, and just keep playing for a while and see if the crash recurs. If it does recur, you knoose scripts probably are safe, and you can use the "disable half" method of debugging for the rest. Essentially, disable half your scripts. See if the error recurs. If not, reenable them, and disable the other half. Repeat until the error recurs. When it recurs you know that the error is in that half of your scripts. Make a note of those scripts, then disable half of them. Then the other half. And keep repeating till you get the error again. Then you know the error is in that half of the scripts (now a quarter), and halve them again and repeat. Eventually (if its a single script problem) you'll narrow down to one script which has the problem, The tricky part is that some errors are caused by interactions between more than one script - but if your code in each script is self contained this will eventually pinpoint the source of the error. If you don't want to disable scripts during a session, disable them after a session ends and reenable them at the start of the next, and just hope whatever is causing the crashes while you aren't playing recurs. If you do this you will have to cycle through each half until the error happens - it may take a while, since the scripts wont be being called as often when people aren't playing. Anyway, it's going to be a long thankless process. Good luck!

aisforanagrams said: Well, all i know is that in my experience with roll20's api system, every game where i use even couple of API codes crash the sandbox with relative frequency.&nbsp; And honestly restarting is easy enough that the effort to debug is much less appealing to me.&nbsp; The point is, there's little guarantee that any QA effort goes into scripts that are accepted to their library and I'm honestly not about to do it for all the scripts I use. I agree there's likely to be a lot of scripts in the library that have bugs that can cause crashes. I know I've seen plenty of reports on the forums where there've been crashes, people have identified the fault with a script, and the crashes stopped. But this isnt a problem with the fundamentals of the roll20 API sandbox.

Well Roll20's API sandbox system from what it looks like does a kind of direct code injection and isn't truly modular in design which makes it more prone to brittleness (and also more difficult to debug).&nbsp; At least that's what it looks like to me. EDIT: again, it's easy enough to restart the sandbox so it's not too much of a hassle to me as an enduser so long as i remember to restart the sandbox on the regular.&nbsp; I appreciate everyone's effort in making people's life easier using the app, but Roll20's framework for API isn't what I would call robust.

I put together ActivityTracker , an event tracking script, that could help narrow down what is happening. The idea would be you can set it up to record some set of events (or even all events), and it will track those to the state. It can tell you when a player connects and whether s/he issued commands/messages, etc. Then, if you see a particular event is breaking your sandbox (the last event before you reboot your sandbox and get an 'on:ready' event), you can at least start to hunt down which of your scripts are trying to answer that event. For instance, you arm the tracker, and walk away from your game. When you return, you see the error, and you reboot your sandbox. Looking at the log, you see a change:token event right before your 'ready' event. You can compare the old token settings to the new, and you can start to look in your scripts for those that answer the change:token event.