[Script] Search -- Full text search for GMs and Players with respect for permissions and many query options!

Search Search provides full text searching across handouts and characters (including attributes). Search uses an Okapi BM25F+ search engine with Porter stemming and stop word removal to provide fast results after building initial indexes. Index construction occurs at API startup, in the background. The GM is notified when indexing is finished. Indexing can take a few minutes, but won't slow down any other processes due to a great deal of deferring of processing. Search respects permissions on handouts and characters. GMs can search on all things, where as players can only search based on what they can see. Seeing a handout allows a player to search on it's name and notes fields. Seeing a character allows a player to search on it's name and bio fields. Controlling a character allows searching on it's attribute s as well. Only GMs can search on gmnotes . Commands !search --help|--reindex|&lt;query&gt; The interface to searching. --help -- Displays this help in game. --reindex -- Rebuilds the search index. This can take a minute or two and is only necessary when you've made changes (like permissions) (GM Only). &lt;query&gt; -- a collection of terms and modifiers to define what is important. See Search Query below for full details. Search Query At it's simplest, a Search Query is just a list of words you're interested in: !search sun blade Terms There are 6 basic ways to enter terms: term -- Just a bare word will rank as normal. +term -- Prepending with + marks a word as important , and it will be weighted double. -term -- Prepending with - marks a word as detracting , and it's weight will be subtracted, lowering the document's score. *term -- Prepending with * marks a word as required , and a document will be rejected if it doesn't have a match. ~term -- Prepending with ~ marks a word as prohibited , and a document with a match will be rejected. term^N -- Appending a ^ with a number will multiply the weight of a term by that number. Numbers can be proceeded by a - to make the term detract, and may have a decimal place. The leading zero may be omitted. Here is an example using each modifier: !search knight *king ~queen +prince -princess chamberlain^1.1 maid^-.3 This will search for knight , require a match for king , reject any document that matches queen , reduce the score of for matching princess , rank chamberlain 1.1 times higher than normal and reduce the score for matching maid at about 1/3 the weight. Prepended modifiers can also be combine with direct weighting: !search *king^.5 throne Must match king , but at half it's relative weight. throne is weighted normally. Fields You can specify search terms to apply only to a specific field using the in:&lt;field&gt; syntax: !search in:bio parents Searches for parents in just the bio field. The in:&lt;field&gt; syntax only applies to the next term listed. !search in:bio parents dragon Searches for parents in jut the bio field, and dragon in any field. If you want to search for multiple terms in a field, you can surround them in parenthsis: !search in:bio ( parents siblings brother sister ) Search for parents , siblings , brother , and sister in the bio field. Anywhere you place terms, you can apply modifiers: !search in:bio ( parents siblings +brother sister^3 ) Additionally, you can apply the same modifiers to the field name to apply them to all the specified terms: !search in:bio^2 ( brother sister ) dragon Both sister and brother in the bio field are weighted double what dragon is in all fields. You can specify different weighting for terms in a field and out of it: !search in:bio ( +dragon ) dragon The most spcific rule is applied. dragon in bio will be ranked double what dragon elsewhere is. If you modify the field and the term , they are combined in the ways you would expected: !search in:+bio ( +dragon wyvern ) dragon wyvern dragon in bio is x4, wyvern in bio is x2, dragon and wyvern elsewhere are normally weighted. Categories You can specify search terms to apply only to a specific category using the on:&lt;category&gt; syntax: !search on:handout gemstone Search for gemstone in all fields of handout s. Just like with in:&lt;field&gt; , you can use parenthesis to group terms: !search on:character( knight paladin ) Search for knight and paladin in all fields of character s. Also like field s, the category can be modified just like term s and passes the modifications down the same way. !search on:character^1.2( knight paladin ) knight paladin knight and paladin are ranked x1.2 when on a character . You can nest in:&lt;field&gt; rules in on:&lt;category&gt; rules. !search on:character^1.3( in:name +dragon dragon ) dragon dragon in character name is worth x2.6, elsewhere on character is worth x1.2, and anywhere else is worth normal. You can get as complicated as you like with this. !search on:character( in:name( dragon -wyvern ~snake) +snake in:gmnotes^3 secret ) on:handout( *treasure *gold ) Search for characters with dragon , preverably not wyvern , and flat out not metion of snake in the name, but rank higher for snake elsewhere and secret in gmnotes is x3. Also include any handouts that mention both treasure and gold . Results The results are sorted based on the combination of the weights of all the terms they matched, then the top 30 are whispered to the user in the chat. Each result row contains a circle with an H for handout or C for character, followed by the name of the matching document with the folder it is in in smaller lettering above it. The name of the document is a link that will open it. At the top right is the total score for the document. Along the bottom of each row are the terms the document matched, with the number of unique fields it matched in, followed by the portion of the score it contributed. This is followed by a list of fields with the number of unique terms matched and the contribution to the score. Looking at these details can help you fine tune your search. Building the indexes in Storm King's Thunder takes around 96 seconds. Building the indexes in Curse of Strahd takes around 66 seconds. Searches are nearly instantaneous! Github: <a href="https://github.com/shdwjk/Roll20API/blob/master/Se" rel="nofollow">https://github.com/shdwjk/Roll20API/blob/master/Se</a>... Support my work on If you use my scripts, want to contribute, and have the spare bucks to do so , go right ahead. However, please don't feel like you must contribute just to use them! I'd much rather have happy Roll20 users armed with my scripts than people not using them out of some sense of shame. Use them and be happy, completely guilt-free! Disclaimer: This Patreon campaign is not affiliated with Roll20; as such, contributions are voluntary and Roll20 cannot provide support or refunds for contributions.

Woah, that's a powerful tool! Awesome!

=D &nbsp;I've got a few improvements I'd like to make at some point, but this is good enough for an initial release. =D

Damn Aaron

Scott C. said: Damn Aaron Indeed! &nbsp;:)

Sweet!

=D &nbsp;I'm really curious to get feedback. &nbsp;There are various tunable parameters to the ranking function, which I'll expose through a settings page at some point. &nbsp;I'm also curious to hear how the attribute search affects things. &nbsp;Right now I use a sum of weights and the same settings for all types of fields, but I might want to change to some kind of weighted average over fields of the same type (attributes, for example) and maybe use different parameters when ranking different fields. &nbsp; &nbsp;It may turn out that it doesn't make too much difference, in the long run, particularly with the ability to weight terms on the field level. =D

A a r oooooooooo n

Do I take it this will get the Vince Seal of Approval? &nbsp;=D Not sure I drew much attention to it, but this gives search capability to the players, who have lacked it in the past. =D &nbsp;I think that can be pretty important, depending on how much data you like to keep about in your game's handouts and visible PCs. =D &nbsp;I know I'll be using it if I can get BGB to install it... =D

Awesome. But now my players will be able to search for rumour information that I have craftily spread across multiple handouts and NPC sheets. Wait, but so can I.

But they'll still have to weight it appropriately. =D &nbsp;

I love this already and I haven't installed it yet. &nbsp;I always wondered why this functionality didn't exist. But I never posted anything in the Suggestion Graveyard. &nbsp;No point. edit: for grammar

Enjoy!

Doug E. said: Awesome. But now my players will be able to search for rumour information that I have craftily spread across multiple handouts and NPC sheets. Wait, but so can I. I would love it if my players did that...

Am I right in assuming that players won't get results from things that they have no access to normally? Would be a shame for them to find my 'End Game' handout that details the BBEG's whole plot and round by round action plan lol.

Correct! Only GMs get search results on gmnotes. &nbsp;GMs also get search results for everything. :) Players get search results based on what they have permission on: Handouts -- if they are in the "in player journals" or "controlled by" or either has "all" in it, they get search results for the name and notes.&nbsp; Characters -- if they are in the "in player journals" or "all" is, they can search the name and bio. If they are in "controlled by" or "all" is, they can search in attributes.&nbsp;

Aaron Don't know if this is the feedback you're looking for, but Curse of Strahd +~100 extra handouts making up a loose PHB and Player Resources gets you. "Load Corpus: 107848/107848... (155.64 seconds)" "Updating IDF values for 54637 terms." "Corpus ready after 155.78 seconds."

Good info, to be sure. (bet you're using the Shaped Sheet?) Generally, I'd like to know: Did it find what you wanted to find? Is the relevancy appropriate or is it muddled by the volume of attributes? &nbsp; Is there a weighting/searching scenario that you aren't able to (easily) setup? Do you notice any performance problems when using it? All that will help me make it better. =D

The Aaron said: Good info, to be sure. (bet you're using the Shaped Sheet?) Correct. I will keep your other suggestions in mind as I use it.

Is there a way to search for words in a specif order? Something like "fire knight" instead of "fire" and "knight?" This is a great way to cut down on the material returned in a search

There isn't. &nbsp;The algorithm I'm using combines the relevancy scores for individual terms which are arrived at by indexes on those terms. &nbsp;The indexing doesn't preserve order, as doing so would drastically increase the size of the index. &nbsp;There are potentially some enhancements I could make around using the indices to pare down to a short list of documents, then do a more intensive sort of matching against them. &nbsp;(In your example, find all the documents with "fire" and "knight" then look for cases where the phrase "fire knight" occurs, or where "fire" precedes "knight" (possibly with a variable relative distance), etc. &nbsp;) If there's enough interest in that, I can certainly put some greater thought and effort into getting it setup. &nbsp;For now, something like: !search on:character *fire *knight^3 is how I'd try to find "fire knight" characters. &nbsp;Or omit the on:character to search for broader search.

I doubt there would be a lot of interest unless you have a bunch of &nbsp;researchers in the crowd. I wouldn't have known how much easier it was till I used some software that employed the technique. Being able to search for word combinations, or at least words in close proximity to each other, can make a search engine incredibly effective.&nbsp;

Totally. &nbsp;Phrases would be great though, and I can imagine people searching for "magic missile" or "death knight" and wanting more relevant terms. &nbsp;Probably something along those lines would be good...