Searching the Forum [tech]

*** Dave H. said: Mike said: (Now if there was only something that tracked how many times I started typing a post only to stop and say, "eh, no one wants to read this" and stopped.  Actually, I'd rather not like to think about how big that number is.) Well, FWIW, there hasn't been anything you have written that I didn't enjoy reading. So either your judgment in sorting out what others want to read is uncannily accurate, or you need to post more of those writings. :-) Considering I just got out of a one-on-one at work where the only criticism boiled down to "You're exceeding all expectations and producing some of the best work out of the whole team, but if you had even a little confidence in yourself you'd be unstoppable," I can imagine it is probably the second one. (though I could see there being a case for the first as well :P)

Well, there's only one way to find out. We'll let you know (gently) if your writing quality takes a nose dive.

Mike said: "You're exceeding all expectations and producing some of the best work out of the whole team, but if you had even a little confidence in yourself you'd be unstoppable"  This is my assessment of you in game as well. If you ever feel like posting something, and hesitate because you're not sure people want to read it, please come back and revisit this post, because I do want to read it. Don't feel obligated to post when you don't feel up to it, as usual. :)

And none of this desire to read more stuff lets you off the hook for artwork, either. Just to be clear. :-D

I finally added a thing to parse dates on individual posts, and once again Tableau's power to snap visualize data helps me out. This is all "substantial" posts (defined as having 1024 characters or more), graphed over time by week. This isn't a bad proxy for activity/interest, and read that way it looks like we're humming steadily along. There was obviously a lot of shenanigans in the December-January timeframe. The recent peaks and valleys are probably cliff-hanger weeks where there's not much to do. How about the meat and drink of our forum novellas, the Cutscene, RP, and Recap tags? Ho. Lee. Shit. The spike in recap posts probably corresponds to when Dave started writing the IC recaps for Jason and Alycia. We've also had a low but steady rate of RP sessions.

One last code change before I go to bed. There's now an Image tag that's attached to any post with no actual text (that is, it's an image-only post). Later on, I'm going to split it into two tags, Image (all posts containing any image link) and ImageOnly (posts containing only an image). I used this to go back and edit my posts, adding LeoMeetsTheInternet and SummerComic tags where I could find them.

Yes!  Finally, a statistic I might lead in... wait no, Dave posts like 8+ relevant images in each of the recap threads.  Dammit, I'm not going to lead in any of these posting trends. In all seriousness, thanks for all the work Bill.  I can imagine this helping out a lot (especially once the Art Thread bloats a couple more pages).

The way that you lead is just difficult for my text-driven tools to measure.

After figuring out what's needed to parse image vs. image-only posts in roll20, here's me and their forum devs:

More forum searching shenanigans. I added Google Analytics tags to the site! It doesn't track what searches you do (and won't ever), but does track visits. We're not exactly a highly active site. :)  But it's helping me with page load and performance tracking. The crawler now gets a list of all image URLs in all posts, and stores that. This means I could (for example) mirror all the in-forum images somewhere else, e.g. into a specialty Pinterest board or whatever. But for now, we'll content ourselves with some statistics. There's 1701 unique image links total in the entire forum. Of those, 888 (52%) are 'embedded' - someone uploaded the image to Roll20, and Roll20 in turn sent it to S3 for storage. The other 813 are off-site links, reaction GIFs, whatever. The offsite links come from 396 distinct hosts. The top hosts by count are Giphy (138), Pinterest (73), Imgur (46), YouTube (26), Wikia (24), Tumblr (20), Tenor (18), and our own wiki (11).

Nothing surprising in those stats, but still interesting to see them. Thanks!

The initial search page + data should load in about 3 seconds instead of 11 now. This is a weird setup, so if anyone is using the forum search and suddenly has problems with it not working, please let me know. EDIT: yeah, every time I crawl the site, one or two pages don't make it. It's not retrying on error due to shenanigans, so I'll fix that over this weekend.

Search hasn't updated in several days because Roll20 made some changes that broke me. I'm getting socket timeouts, and my URL de-duping isn't working thanks to their new pagination scheme. I'm working on fixing it.

FINALLY. Indexes were rebuilt: 2018-9-30 13:18:46 I switched from the built-in skip duplicates logic (which is shit) to seenreq (which is frustrating), and used async/await instead of promises for better troubleshooting. But it's working now, and I verified that it's retrying all errors, and grabs each page only once.

Thanks, Bill! Your computer-fu is deeply appreciated.