API Dev Server update 10/17: Sheet Worker Support

Hello all, Today we've rolled out a new version of the API that's currently only active for DEV SERVER games. In particular, it adds the new function: setWithWorker(properties) This function works just like the current set() function, except when called on an Attribute object it will also cause the relevant sheet worker code to be executed by the API server, just as if a player had changed the value on the client using the character sheet. Example: getObj("attribute", "-KUI1fO2L7Jv0Y4AOSFK").setWithWorker({ current: "Cleric" }); In addition to that, you can use the new function onSheetWorkerCompleted(function() {   //do something here }); to register code that will only run after all sheet worker execution has finished for the current 'stack' of sheet worker calls. You can register the callback before you call the setWithWorker() function. Note that each callback you register will only run one time. I've tested this code with the OGL 5E sheet and it should all be working fine at this point. If you notice any issues with other sheets where the API isn't behaving the same as a client, please let me know. Thanks!

It should be: attributeTrackedObj.setWithWorker({current: "test"}); Where attributeTrackedObj is an object you got with e.g. getObj("attribute", "-ABC123") or findObjs. Here's the example I'm running in my test game this is working currently: getObj("attribute", "-KUI1fO2L7Jv0Y4AOSFK").setWithWorker({    current: "Cleric" });

Also just to be sure you should be seeing a new log showing up in the API console when it starts up that says: "Found webworker script"

Lucian said: Fantastic - many thanks - this is going to make a lot of things much easier. One question - what does the onSheetWorkerCompleted callback relate to? Is it just the next completed set of sheetworkers, or is it all future ones? It's just the next set of completed ones. So each callback you register will only ever get called one time.

Lucian said: Another observation - it seems like the onSheetWorkerCompleted callback is called after the current event processing completes irrespective of whether any calls to setWithWorkers were actually made. I guess it should be possible to work round this, but it does potentially make life a bit more complicated for script authors. If it's hard to suppress the call because of how it's implemented, could we perhaps have the callback accepting a parameter that indicates whether any workers were actually executed? So I actually had to change the way this worked because originally when you added a callback even if you called setWithWorker() if a) there were no events triggered by the change or b) if the data you were setting wasn't different, the callback would never fire. You're saying in that situation you wouldn't want it to fire and instead it should stay on the stack and wait for the next time something does actually change?

Lucian said: It looks like sheetworkers triggered this way don't trigger API events.  I can see why that's probably desirable in most cases - it certainly avoids a lot of potential for evil cycles etc - but it does mean that the behaviour is inconsistent with what happens when the client makes the same change. I can see that causing grief for people as well, since any scripts that rely on being able to monitor the value of an attribute are going to get blindsided if another script changes them indirectly in this way. Could this perhaps be an optional behaviour, or would that be really hard? Yeah this is one of the weird places where things intersect. You're right that on the client if a sheetworker changed something it would trigger an API event because it would be seen as the "player" changing something. However the sheetworker modifications are considered to be done by whatever originally triggered them, so in this case the "API" is changing things, and currently the API does not self-trigger change: events whenever it changes something itself. I could possibly make a change so that we could force those API events to fire but as you said I think it opens up a lot of potential for people to get in bad infinite loops since in a lot of cases the API script author may not be completely familiar with everything the sheet worker is going to do...

I will go through and answer your questions more thoroughly tomorrow when I'm more around, but for now I'm going to push out an update that should cause the log() function called from inside the sheet worker to output to the API console which should help with your testing.

Okay, so the logging should be coming through now. It's not as complete as client-side logging since it doesn't support sending whole objects over the log obviously, but hopefully that will help. Also something else I noticed to keep in mind, there is no sheet:opened event that is triggered ever on the API side. I don't know if that affects anything regarding your import stuff but just something to note.

Jakob said: Being able to fire "sheet:opened" from the API would be great for a 'please update all my characters in this campaign to the newest sheet version' button. My concern with doing that via the API is just that it then precludes it for anyone who's not a Pro. Also, I'm not sure what sort of load that might put on the servers (updating like 100's of characters in a campaign all at once). That's the same reason we haven't put a button like that on the client side, either. But yeah, we'll definitely consider adding it to the client.

Lucian said: Is there any way that we can debug sheet workers on the API server? I think the most convenient option would be to have console.log bound to the existing API log method, so that pre-existing logging output in the sheetworkers can end up in the API log. For extra Amazing Points having some option to show which sheetworker handlers were being triggered as they were run would be extremely handy... How does ordering/timing work? Do sheetworker events get queued in the order that the setWithWorker calls were made and then executed serially after the main API thread returns control back the Roll20 framework? I'm guessing that secondary sheetworker events triggered by the first batch of sheetworker updates may happen in any order, depending on the order in which responses to the first set of getAttrs/setAttrs arrive? 1. So we added that already, hopefully that's helping. 2. Each setWithWorker() call immediately triggers all relevant events to happen inside the sheetWorker. The sheetWorker runs asynchronously though (just like on the client), so there's no guarantee that each sheetWorker event will complete before you do your next setWithWorker() call. There's only one sheetWorker thread though (just like on the client) so it's not like setting/getting is happening in parallel, it's just asynchronous. The setWithWorker() call should be equivalent to changing something on the character sheet on the client and then clicking off the field, which on the client immediately saves that one value and then fires off the sheetWorker events. On the second part, yes the second batch may happen in any order. However, again, this is exactly how it works on the client. The only thing I can think that might be a "gotcha" is that on the client I expect most people fill out the sheet rather slowly (click on a field, then wait to see what changes, then click on the next field)...if you're firing off a bunch of setWithWorker() calls back-to-back on the API server that would be happening much more quickly than on the client. You can nest setWithWorker() calls inside of onSheetWoerkerCompleted() callbacks if you want to go more "slowly." Lucian said: Ok, still doing my best to debug this - proving quite challenging without any logging - but I've found another issue that will need some sort of solution - getTranslationByKey doesn't work on the API server. This is problematic for the Shaped Sheet which uses this to be able to have a dynamic skill list with localised text available for the defaults. Can we get the translations loaded into the API server as well somehow? Yes I will add those, I didn't know if it would be needed or not but I should be able to get it in. Might take a few days, though. Lucian said: Without being able to see more of the internals I'm stabbing in the dark here, but it looks like attribute creation (perhaps only in repeating sections?) isn't triggering things the way I'd expect, only attribute modification. Feel free to PM me if you want me to show you the specific tests I've been running in more detail. I'm not sure if you gleaned anything further after the logging was put in place, but assuming this was still an issue I tried putting in some code to hopefully handle it better. The same setWithWorker() stuff should in theory be triggering on the creation of a new attribute as updating an existing attribute, but it's possible it wasn't working due to the way the API handles new object creation. Anyway, let me know if that helped. The Aaron said: What about calling the callback with some sort of context object?  Say something like: { workersExecuted: true, attributesChanged: [ 'foo','bar','repeating_inventory_-Jasd123fas_qty' ] } I added a new parameter to the callback, so you can now use it like so: onSheetWorkersCompleted(function(context) { //context.workersExecuted will be 'true' if sheet workers fired since the callback was set, 'false' otherwise. }); Okay, well hope that all helps! Let me know as you come across more stuff.

Lucian said: In the Shaped Sheet traits, actions etc are parsed from an attribute called content_srd, which is the droptarget for the Content field from the compendium. If I set this field to a value that matches the format from the compendium, the sheetworkers parse out the individual traits and actions from the content_srd value, and create new repeating rows for the relevant actions and traits. Each row gets a name value populated, and a "freetext" which contains the complete SRD text of the trait/action. Also, what's the current script you're working on that uses this functionality? Just curious. One of my main concerns with this (and the API in general) is basically people creating some sort of "one-click script" that ends up blowing up game sizes to something really really unreasonable. So just to be sure -- you're not making a script that like imports the whole Compendium into someone's game are you? Haha... 

Kryx said: The script can only import 1 creature at a time and there are no current plans to change that. I doubt we would. The import was designed to mirror to srd import to reduce complexity and code. It fulfills the same function except imports from a user created json file. Yeah okay that sounds fine. Basically a Compendium-style import for custom/homebrew creatures. Cool. I would just be sure to point out to people that the more Monsters you load your game up with, the larger the game file gets and that's bad because a) it takes a lot longer to load, and b) it can cause issues if it gets super large (we have some people who can't even create copies of their games right now because they have put like 400+ Characters into them). So I think if it's a situation where using the API allows you to not have to put everything into your game right away and you can just pull it in as-needed, that's a great option. I'm actually thinking of putting something on the Campaign details page where people can see the actual "size" of their game's data file just so they know how large it is, sort of as a community education thing.

Scott C. said: Not related to this, but to the previous API update. Any ETA on when noarchive will be fixed? It still isn't working after the last update (although the error is now more sporadic). Bug Report:&nbsp; <a href="https://app.roll20.net/forum/post/4093752/api-bug-noarchive-equals-true-still-funky/?pageforid=4093752#post-4093752" rel="nofollow">https://app.roll20.net/forum/post/4093752/api-bug-noarchive-equals-true-still-funky/?pageforid=4093752#post-4093752</a> It's on my list, I don't have any further info at this time, I'll update that thread when I do.

Okay so I'm not really able to get a test case for this yet. I realize it's happening with the sheet worker that you're using but as you noted, there's so much going on with that it could really be anything. Here's a simple test case I put together that shows that at least with simple scripts, having the API create a new attribute and then use setWithWorkers correctly triggers the sheet worker to run in the API, and if that sheet worker script that the API is executing then subsequently creates a new attribute, the sheet worker hears the change event for that attribute. <a href="https://gist.github.com/rileydutton/b988f2e1d05ff625674f89e39ee732b8" rel="nofollow">https://gist.github.com/rileydutton/b988f2e1d05ff625674f89e39ee732b8</a> Can you modify that test case in your own testing to get it to show the behavior you're describing (where there's a 'break' in the chain?) If so that would make fixing it go a lot faster.

I've also just pushed out an update that adds support for using getTranslationByKey() along with loading all the relevant translation strings into the sheet worker on the API. However, note that since the API is run at a global level, the language is currently always set to "en" (US-English). I'm not really sure how we'll end up handling this long-term, it might have to be something we add to the API Settings page for the game or something like that.

Lucian said: Ok, thanks Riley. I will try and modify the testcase you've created to get it to break in the way I'm seeing - might be a day or two but I'll see what I can do.&nbsp; Sounds good, I'll keep an eye out :-)

Lucian said: Ok, that was a lot easier than I was expecting. It looks like it is a repeating section issue. My modified gist is here: <a href="https://gist.github.com/symposion/a1d5ef45629170de" rel="nofollow">https://gist.github.com/symposion/a1d5ef45629170de</a>... You'll see that if you edit the base attribute manually from the character sheet, you get a new repeating section row with both values filled in; but if you trigger it with a chat message, only one of the values in the row is filled in - the secondary sheet worker triggered from the creation of the first repeating section attribute is never fired. Great, thanks, I can fix that :-) I'll let you know when it's fixed, might be Friday before I get a chance to get to it.

Okay I believe I have this fixed. Let me know if you run across any more oddities. Thanks!

Let me know what you find with the getTranslationByKey() stuff. At this point the function should be fully working so I don't know why the key not found errors would be happening...maybe make sure its not always happening with the same key? For the first part, the way that the callback handler works is very basic. It just has a "listener" that hears any "traffic" between the sheet worker thread and the parent thread and anytime any traffic happens, the timer gets reset. Currently the timer is just set to 250ms though, so I can try upping that to 1,000ms and see if that helps. But the timer is getting reset on any call to getAttrs() or setAttrs(). So if it's expiring faster than you think it should that probably means there is a delay due to some actual processing that is happening on the sheet worker thread (i.e. it's doing a big calculation)...I can increase the time like I said but again, this is not some really elegant solution or anything (as I pointed out when I put it in)...the only way to be extra-extra sure that something happens before something else would be to add an event listener for the specific variable that you are waiting on being set.