How to Store and Retrieve large data tables without massive character sheet bloat

Hello all, I am working on a sheet for a space-based game that has MANY possible ships with a pretty lengthy stat set for each ship. And also uses several data tables to determine weapon ranges, bonuses, damage locations, etc. I would really like to be able to give players dropdowns for this - filter by race, filter by ship, and then have their sheet auto-populated with all ship stats as it can be pretty tedious to do by hand every time. So, I guess this is similar to a stat block import - but I want it to be driven from within the sheet without a copy/paste. I would also LOVE to be able to do this without storing the data IN the sheet, so that new things can be added without requiring a new sheet upload, but I understand that may be impossible. That being said, I also don't want to bloat the HTML/JS with all of this data if I can help it - the lookup tables for range, damage, etc, I am less worried about, but the ship "database" makes me a lil nervous. How do you handle storing and accessing large amounts of data without having your own version of and integration with the Compendium? Is there an API method for this? I am an experienced developer but totally new to Roll20 sheet creation. Some of the limitations are infuriating but I am learning to work within what they allow. Any guidance for something like this would be most welcome. Thanks!

Just use sheetworkers, and have your tables set up in there. The sheet's script is only loaded once per client browser since it's identical on every character sheet, so bloat isn't really an issue beyond maybe a few extra milliseconds on Campaign load. The 5e sheet's script is around 18.5K lines long, for example. It's only sheetworkers which constantly trigger (particularly calling setAttrs) that cause lag in the game - the size of the script block doesn't matter. You will probably need to update the sheet to add more data, though. The only way around that is either JSON parsing from a textarea, or writing your own mod parsing system so custom types can be added. No idea what your game looks like, but something like this: ship hull: large; small weapon mounts: 12; medium weapon mounts: 10; shield generator: 5; engine type: 9.5 with an action button underneath. This kind of thing is probably only appropriate for relatively simple stuff though, or it'll be nightmare to code. If the stuff you want to add is really complex, you're probably best off just doing a sheet update.

Yeah, my thought was to embed it into the JS section (if no other options will work) and try to make the data as compact as possible, a CSV even vs JSON - space over readability. And try to encode certain things only once - all engines of a certain type have the same stats, I only need to store that in an "engine" CSV and then store the index in the ship CSV to reference the data, same for weapons and shields. This is really not how I'd like to do it but given the limitations maybe it's the only option. Oh and one other thing - is there a way to add items to a repeating section (weapons, engines, shields) from a sheetworker? So if I load in 4 weapons how would I make a section for each of them and set the stats since you can't alter the DOM? Thanks!

Yeah I wouldn't worry too much about space, I'd just go for whatever is the most efficient. If you're planning on adding or editing later, I wouldn't underestimate the importance of readability either. Even if your data tables are 10,000 lines long, that's still very little data compared to a hi-res map graphic and a 10 minute background track. The client only grabs the character sheet data once on Campaign load, after that it's all referenced in the local browser instance. You can potentially look at other sources for your tables using the API, but the character sheet sandbox can only interact with the character sheet (actually, it can read from the chat window in a limited context now with Custom Roll Parsing). But embedded in the JS is going to be the most straight-forward way. Is there any particular reason you have an aversion to storing it in the main code? For repeating sections, you just use setAttrs with the full repeating attribute name, including rowID. I would use a helper to do this, so my main functions are only dealing with the suffixes and values. As a simple example:     // Use option {existingId: <id>} to use an existing rowId     const createRepRow = ( section , attrs , options ={}) => {         let output = {};         let rowId = checkRowId ( options . existingId ) ? options . existingId : generateRowID ();         for ( let key in attrs ) {             if ( attrs [ key ] != null ) output [ `repeating_ ${ section } _ ${ rowId } _ ${ key } ` ] = attrs [ key ]|| '' ;         }         return output ;     }     createRepRow ( 'weapons' , { name : 'Stick' , damage : '2d6' , bonus : '2' });     // Expected output:     //  {     //      repeating_weapons_-1234567890abcdefghi_name: 'Stick',     //      repeating_weapons_-1234567890abcdefghi_damage: '2d6',     //      repeating_weapons_-1234567890abcdefghi_bonus: '2',     //  } createRepRow returns an object that's ready to plug in to setAttrs to create the row with the starting values. You can certainly add more parameters to the function if you want to add more capability, or approach it another way if you prefer (like using Arrays for input instead of an Object), but that's the general idea. generateRowId() is an inbuilt sheetworker function, and checkRowId is another helper that validates the UID to make sure no bogus attributes are created

Oosh, in your code sample above: let rowId = checkRowId(options.existingId) ? options.existingId : generateRowID(); Is checkRowId a function you havent included here?

Thank you! I guess my fear comes from just feeling like it's "wrong" to store the data that way, in any traditional "app" you'd use a database (or any other dedicated mechanism to do it), but I suppose you're right, in the end, it's a few thousand lines, which is a "lot" but if all that really means is 15-20k of data, it's pretty trivial in the overall scheme of things. I really hoped for the ability to grow the data without requiring a new sheet to be uploaded but I'll just deal with it as it is for now. And thank you for the helper method for repeating sections, I had no idea if I'd be able to add those when importing and setting up the data for the ships, I'll play with this all a bit over the weekend and see how it goes!

On the topic of expanding the data, you could include a separate input/textarea for pasting a big JSON of the data. Unfortunately, it would need to be pasted separately for each character in use, which is one reason it's handy to have it embedded in the JS of the sheet. If you're creating a sheet for general use, it's best to use to use the sheet JS. If you're creating a sheet for your own group's use, and intend to keep Pro account for duration of your campaign, it might be better to create an API script to manage the data. Have the character sheet layout set up so that it can display the data, and use an API script to hold your data set, create a macro that lets people make selections and trigger the script, which then pastes the data to the sheet.

I think the second of those is more along the lines of what I was thinking and how I'd like to do it, but there may be a point down the line where I'd like to release it as a general sheet so I think I'll just commit to getting the data into the JS directly. I'll probably end up building a tool to export from a google sheet or excel and build the JSON/CSV/whatever so I can just add that into the script easily but for now I just need a few entires to test with and make sure I can actually populate the sheet as I intend. Then I'll worry about building the "database" and making it easy to pull in. It's good to know that there is an option to do it with the data "externally" I just wish it were more accessible to do so. Thank you all for the help!

GiGs said: Oosh, in your code sample above: let rowId = checkRowId(options.existingId) ? options.existingId : generateRowID(); Is checkRowId a function you havent included here? Yep, I did edit in that it's another helper function - I was too lazy to delete the reference :) Also Chuck, one other option which is a bit clunky/experimental - you can use the new(ish) Custom Roll Parsing to do a bit of acrobatics with the character sheet sandbox. To cut a long and technical story very short, you can potentially have a "Mods" character sheet with a preset name that you could use for expanding your game options. Similar to the textarea=>JSON idea, only it would be a single character sheet that all other sheets can refer to for the expanded material. That might be easier than pasting stuff into every sheet. It's still bound by the same limitations (e.g. you need perfect syntax to JSON.parse() without an error), but at least it's a single, central location that other sheets can refer to. Might be an option down the road if you still want the sheet to be extensible without needing to update.