Data Storage Options

Since my GM is not a developer, I find myself constantly having to look up values and do a lot of API trickery mid-game to accomplish some of the things he needs. For example, we keep historical records of xp awards in a dummy character specificially designed to just store data in it's attributes. Since everything is stored as an array of JSON objects, reading it all as one string becomes a bit overwhelming for him, and I have to extract the necessary record for him and delete or change it if he types something wrong, this leads to a lot of tedium. I have two ways I can go to solve my problem, but it kind of gets into potential limitations of Roll20. The first option is to dump the information into a handout or character notes in table form, instead of using a character attribute. I'm not sure what the maximum length of a handout is, or of an attribute's current value either. This will allow him to see the specific xp award he's looking for, and just delete or change it by providing a numeric id. Without knowing the maximum length of the handouts or notes sections, I might run into a problem as the campaign progresses. The second option is to fully implement an alternate character sheet that changes styles (hiding the original character sheet) to reveal a GM-only interface on our dummy character that would allow him to customize these items with a graphical interface and would also separate the values from all being one attribute to being a collection of attributes. It gets over any potential maximum length issues from the first option, but could also dramatically increase the size of the campaign file, as well as requiring significantly more work and making our character sheet obscenely huge (our current custom character sheet is about 6900 lines of HTML and 1100 CSS). I'd prefer to go with option 1, as it requires much less work on my part. However I need to make sure that whatever solution I implement will work throughout the entire campaign. The XP Award history is the largest portion of what I need to store by a wide margin. We don't store much other historical data aside from that. Any advice you guys have on Roll20's limitations and/or how this has been accomplished before would be appreciated.

My first instinct is to as you for more details on the problem you're trying to solve, rather than the solutions you're proposing. There are probably some solutions you haven't considered, and fully specifying the actual problem will help others (and even you) take a step back and consider what might fix it. From your description, it sounds like your GM wants to keep track of a list of XP rewards (by character? with notes?). Your GM wants to be able to review those rewards (by character? by time? by search of notes?), and make changes to them (delete? change amount? update note?) Have I missed anything?

Right now, xp awards are stored as a JSON object: { 'id': 0, 'amount': 0, 'characterid': '', 'reason':'', 'processed':0 } Every time the GM awards experience, it adds another one of those objects to an array. At the end of the session or adventure, he runs the !processXP command, which goes through the array and adds up the value of all awards where processed == 0, adds the value to the character's xp, and then changes the processed value to 1. If he removes a processed award, it simply recalculates the value of all items where processed == 1, and ignores unprocessed results. Historical data is kept so the GM can add, edit, or delete awards, as well as check counts of specific awards and reasons. For example, seeing how many awards have been earned by a given character for 'Skill Use: Knowledge (Religion)' or for 'Roleplaying', etc. XP is typically awarded automatically, as the GM has access to a '!promptSkill' function, that uses the new API buttons. It throws a button that players can click and, if they have access to one of the characters outlined in the arguments for !promptSkill, it will roll their skill check against the DC set by the GM. If they succeed, they're automatically awarded 25xp. If they fail, they're awarded none. As with just about everything I do that's automated, I try and keep an easy override process in place to make sure the GM always has the ability to undo something done by the scripts. So the GM needs to have a way to perform basic CRUD actions on the collection. Since this can commonly stretch over multiple game sessions, the state object is inadequate, as it is not fully persistent. Any API crash or long period of inactivity could wipe the entire collection. So I need something that could potentially store a large amount of text, as I don't want to get 5 months into the campaign only to have errors start cropping up because a handout can't hold any more xp awards or the campaign has become too large. Also, I have my own db server at home, and can easily implement an mvc web api if there are ways to use ajax calls to send and retrieve data out of Roll20, though I don't think there is.

I feel like I probably use the API more than most people, and I have NEVER lost anything in the state. It's persisted almost immediately to Firebase. If data is going missing from your state object, it's likely the result of a script error somewhere, possibly a misunderstanding of the way Javascript objects work. If not, there is some serious bug with your campaign that needs dev attention. Anyway, I would still recommend writing it to the state object. If you're worried about losing data, I'd implement a backup strategy for the data into something like a handout (base64 encode a JSON object there every 10 minutes or so). It sounds like you're already comfortable with API Buttons, I'd probably build a better interface in the chat for the GM. !xpReview to get buttons for showing the information various ways (all awarded, all unawarded, all by character, etc). For each listing have a command to award or retract them, a button to change the value, a button to set the character, that sort of thing. Maybe partition the data by session (date?), could list the sessions with buttons at various levels to show all the entries at that level. If you still want to go the handout route, I'd suggest that simpler is better. I would avoid HTML tables, as you're going to have to read that info back out, and writing your own HTML parser sounds like a pain in the butt. 1 A "Bob the Enchanter" 25xp Use Skill: investigation 2 X "" 150xp Convince the King to grant clemency. Something easy for your GM to write and modify, something easy for you to parse: /^(\W+)\w+(.)\w+"(.*?)"\w+(\d+)xp\w+(.*)$/ Even doing that, I'd probably still keep a copy of the parsed state, then do comparisons on change, prompting for corrections as needed. There isn't a way to retrieve data via AJAX from the API. That could be great, but also potentially hazardous.

It was my understanding that the state object wasn't persistent. Though my last real interaction with using it was last year. Has it been updated recently? If the state object persists between crashes and inactivity resets, then that seems like the best solution. Also, I already have the necessary function built for reading data into and out of a simple HTML table. I've been using them similar to how HoneyBadger set up the formats for the PowerCards script, where my GM can modify settings using handouts (this also helps him move settings between campaigns). Thanks for your help again. :D

I think way back last year the state might not have been persisted, but as long as I've been actively developing, it has been and has been extremely stable. :). I know HB has expressed some concerns about state in the past, so I gather it was pretty seat of the pants in early days. =D No problem, happy to help!

I just think the state is too fragile to be assigning important information to. All it takes is one little line in any api script and the whole state gets wiped.

Ah, that makes sense. I can see why you would feel that way. Another great thing about writing to handouts is the ability to migrate data between campaigns (though IDs all break, but other stuff would work).

I do write to the state for the powercard formats so it has less of an impact when someone runs a powercard, but the handout is always there to back it up if the script can't find the info in the state.

That's a good policy. I'll have to think about productizing that.... =D

It's also easier to edit the handout than it is to use keyboard commands. :D

Do you guys know how many characters a handout can support?

Charles H. said: Do you guys know how many characters a handout can support? Doesn't matter. Use a handout for each session and stick them in a folder somewhere. :D

Charles H. said: It was my understanding that the state object wasn't persistent. Though my last real interaction with using it was last year. Has it been updated recently? The entire point of the state object is that it's persistent. In about a year of using and writing API scripts which store data in state, I think I've had it become corrupted twice, at which point the API server automatically resets it. The way my scripts are set up these days, a reset state object is going to get rebuilt as soon as the sandbox restarts.

Charles H. said: Do you guys know how many characters a handout can support? Probably 10mb, based on the maximum size of a single value in firebase: <a href="https://www.firebase.com/docs/web/guide/understand" rel="nofollow">https://www.firebase.com/docs/web/guide/understand</a>... I imagine things will get unwieldy for other reasons before you approach that maximum. Sharding across multiple handouts is probably a good idea.

That about sums up all of my questions. You guys rock. Now to rework my scripts.

No problem! =D