API Update 3/20: New NodeJS Version, Sheet Worker Support in API, Speed Improvements

1490020254
Riley D.
Roll20 Development Team
Hey everyone, We've now pushed all of our work that has been on Dev for a few months to the Main API server. That includes: An upgrade to NodeJS v 7.x Sheet Worker support for API scripts  Massive speed improvements especially to findObjs() on large campaigns You can find previous discussion on Dev of all these features here if you want to read about them: https://app.roll20.net/forum/permalink/4507273/ https://app.roll20.net/forum/permalink/4748714/ If you have any questions or run into any bugs with these new features, please let me know in this thread, I'll be closing those previous discussions. Thanks!
1490020452
The Aaron
Pro
API Scripter
Yay!!!
1490041763
Brian
Pro
Sheet Author
API Scripter
Sweet!
1490042005
Scott C.
Pro
API Scripter
Woot
1490044069
Lucian
Pro
API Scripter
Big thumbs up! Thanks for all the hard work nailing all the strange issues!
It seems my favorite scriptesrs are happy, which makes me happy. Does this translate to anything meaningful for muggles like me?
1490059569
The Aaron
Pro
API Scripter
Probably akin to working in a nicer art studio.  The art will be the same, but it will be much more fun to make it. =D
1490077458
Sky
Pro
And more learning for me... -_-
The Aaron said: Probably akin to working in a nicer art studio.  The art will be the same, but it will be much more fun to make it. =D Aha. Then I think a metaphor for me might be: upgrading Photoshop. Same output, but the creative experience is slightly easier/more fun.
1490115473
Brian
Pro
Sheet Author
API Scripter
keithcurtis said: The Aaron said: Probably akin to working in a nicer art studio.  The art will be the same, but it will be much more fun to make it. =D Aha. Then I think a metaphor for me might be: upgrading Photoshop. Same output, but the creative experience is slightly easier/more fun. And as with an upgrade to Photoshop, there are a small  number of brand new things that can be turned out. For example with this upgrade, one ES6 feature it includes is destructing which, among other things, can let us write a function designed to return more than one value. Strictly speaking, the function is just returning an object or array: function example1() { return ['foo', 'bar']; } function example2() { return { foo: 'bar', fizz: 'buzz', }; } But the consumer code of such a function can now quickly split that result up into multiple variables: let [a, b] = example1(); // a === 'foo' // b === 'bar' {foo: a, fizz: b} = example2(); // a === 'bar' // b === 'buzz' let {foo, fizz} = example2(); // foo = 'bar' // fizz = 'buzz' Previously, attempting to get multiple results from a function would require either several statements, or an output parameter (or both): let result = example2(); a = result.foo; b = result.fizz; function example3(out) { out.foo = 'bar'; out.fizz = 'buzz'; } result = {}; example3(result);
1490125066
Stephen L.
Pro
Marketplace Creator
Sheet Author
API Scripter
So many cool new ES6 features!
1490344335
Lucian
Pro
API Scripter
Ok, so I'm getting this error occasionally now: "TypeError: Cannot read property 'replace' of undefined\n at eval (eval at messageHandler (evalmachine.:282:6), :8:2619)\n at Array.forEach (native)\n at Object.process (eval at messageHandler (evalmachine.:282:6), :8:2056)\n at Object.eval [as -KfxtCMhnofbkY--eYIM//false//0.28305818277668915] (eval at messageHandler (evalmachine.:282:6), :89:2823)\n at _fullfillAttrReq (evalmachine.:257:31)\n at messageHandler (evalmachine.:288:6)\n at process. (/home/node/d20-api-server/node_modules/tiny-worker/lib/worker.js:68:55)\n at emitTwo (events.js:106:13)\n at process.emit (events.js:194:7)\n at process.nextTick (internal/child_process.js:766:12)" Is this another "You deleted a character while I was still doing stuff with it" error?
1490366072
Riley D.
Roll20 Development Team
Hmmm that stack trace would seem to indicate that you are trying to call .replace() on an undefined variable (although I assume it was supposed to be a string) inside of your sheet worker code. It's inside of a Array.forEach call inside of a getAttrs() call if that helps. And the character ID it was fetching the attributes for was -KfxtCMhnofbkY--eYIM. 
1490369629
Lucian
Pro
API Scripter
Thanks Riley. A bit of our own non-determinism in our sheetOpened sequence - found it now. Sorry for the false alarm!
1490950038
Lucian
Pro
API Scripter
Hey Riley, I'm seeing a slightly weird thing that may connect to something you previously mentioned as a limitation of the system. I'd like to know if it's expected behaviour, and if so, if there's a recommended workaround. What I've been seeing is that sometimes, the API sheetworker callback happens before the first sheetworker even gets called. After a little digging, it became clear that this was happening when there was another script installed in the campaign that also does a setWithWorker on startup. It seems like maybe my callback is being triggered by the completion of the setWithWorker call in the other script, before any of my workers actually begin execution. To clarify the situation: There are two scripts In their respective on('ready',...) handlers, both scripts create an attribute and call setWithWorker on it The other script doesn't actually register a callback - I suspect that the setWithWorker could be replaced with a .set in this case, but it's not my script Sometimes (perhaps related to the order of 'compilation'/execution?) this results in my callback being called before the first of my sheetworker event handlers is ever called. Is this expected behaviour? Is there something I can do differently to protect myself from this (other than wacking in a startup delay, which is what I've done as a stopgap)
1490950277
Lucian
Pro
API Scripter
Another question unrelated to the issue mentioned in the previous post: is findObjs({type: 'attribute', name:'my_attr_name', characterid }) now basically the same speed as getAttrByName() ? I'd like to know because I'm thinking of replacing getAttrByName with findObjs to work around one half of this: https://app.roll20.net/forum/post/4818851/spurious... - but I don't want to do it if I'm going to take a significant performance hit from doing so.... Cheers
1490972971

Edited 1490973035
Riley D.
Roll20 Development Team
For your first question, I'll have to think if there might be a way. Since the onSheetWorkerCompleted is a global 'lock', there isn't really any way to prevent another script from calling setWithWorker() while your stuff is happening.  That said, is your onSheetWorkerCompleted() registration not happening immediately before your first setWithWorker() call? What it sounds like is that you are calling onSheetWorkerCompleted(), then a lot of time is going by (during which this other script is calling setWithWorker() and the queue is clearing out), then later on you are calling setWithWorker()? I would think if you did onSheetWorkerCompleted(), followed in the same execution thread with a setWithWorker(), that other scripts calling setWithWorker() shouldn't matter... For your second question, findObjs() would actually be faster than getAttrByName() in that case, because getAttrByName() does additional work to check for things like default values, nth number repeating sections, etc. So if you know that the attribute is definitely already going to be set, and you don't care about default values or repeating_$n_whatever, then you can just use findObjs() instead. At its heart getAtrtrByName() is just that exact findObjs() call, just with the extra logic before and after it to handle those other special cases (like the attribute not being set yet but you want the default value instead). var search = findObjs({_type: "attribute", name: attrname, _characterid: character_id}, {caseInsensitive: true});
1491240510
Lucian
Pro
API Scripter
Riley D. said: For your first question, I'll have to think if there might be a way. Since the onSheetWorkerCompleted is a global 'lock', there isn't really any way to prevent another script from calling setWithWorker() while your stuff is happening.  That said, is your onSheetWorkerCompleted() registration not happening immediately before your first setWithWorker() call? What it sounds like is that you are calling onSheetWorkerCompleted(), then a lot of time is going by (during which this other script is calling setWithWorker() and the queue is clearing out), then later on you are calling setWithWorker()? I would think if you did onSheetWorkerCompleted(), followed in the same execution thread with a setWithWorker(), that other scripts calling setWithWorker() shouldn't matter... This is the weird thing. My script is calling onSheetWorkerCompleted; then on the next line, it does setWithWorker; and then it returns out from the on('ready') handler without doing anything else. The next thing that happens as far as my script is concerned, is I get the sheetworker callback - but logging from the sheetworkers themselves shows that none of them have been called yet. As yet I haven't done any hacking around with the other script - will do this  in the next couple of days and add any additional information that i find as a result. It's possible that my diagnosis of what is going on is incorrect. I'm sure that the callback is coming before any sheetworkers are executed; and it does seem like disabling this one other script (which does have setWithWorker calls in it) fixes the problem. But it may be that the setWithWorker calls in the other script aren't happening where I think they are - until I put some extra logging in there I won't know for sure; and it's also possible that the relationship with that other script is a coincidence, and this is just a weird timing issue. Will try and get you some more hard data soon. Thanks for the info on findObjs vs getAttrByName - that's useful.