Scrape chat for use in external apps/integrations

Hi folks, I have seen references to using web scrapers to pull info out of chat for use in external apps or integrations. I'm trying to explore this further, but I my Google-fu is failing and I can't find the threads referencing what people have successfully done. I'm sure I can figure it out given time, but I'd rather "stand on the shoulders of giants" and use other people's work if possible ;) Anyone able to point me in the right direction? Thank you!

This really has nothing to do with the API, unless you’re wanting to dump specific data out to scrape. Either way, the scraping would likely be part of an extension you’d write which would register an event against the chat area which would be called when the box’s contents changed. 

The Aaron said: This really has nothing to do with the API, unless you’re wanting to dump specific data out to scrape. Either way, the scraping would likely be part of an extension you’d write which would register an event against the chat area which would be called when the box’s contents changed.  I struggled to find the right location for this but the blurb for the API forum says "Get Help, Share Scripts, & Discuss API Customization & Automation" and what I'm trying to do is automation... :) Ultimately what you mentioned would be a next step after getting basic scraping working. Something like a listener looking for certain chat output that would trigger an API event that would in turn output formatted text (to me as GM) that the scraper could grab. If the mods want to lock/delete this then so be it, but in lieu of a REST API this is all I can think of :(

Let's try this from another angle. What task are you trying to accomplish? Frequently there are other ways to get to the same destination.

keithcurtis said: Let's try this from another angle. What task are you trying to accomplish? Frequently there are other ways to get to the same destination. Thanks for your reply! There's two things I'm exploring: 1) analytics on dice rolls using Power BI (eg. how many times has player X rolled a 1 or 20 and so on) 2) aggregation of party stats and in game events into a 3rd party tool (right now I'm looking at a couple of different endpoints such as a custom app I'm building on the Salesforce Lightning platform as well as sending some events to discord) For number 2) as a middle ground I'm working on an API script that will at least do some aggregation in Roll20 (as a handout for example), but my ultimate goal is to get the data off-platform.

Hm. That's definitely beyond my meager skill set, and not doable with anything off-the-shelf insofar as I know. There was a project a couple of months ago to build a Twitch overlay that pulled data in a live fashion. You could look for that; it might give you a starting point.

That's very helpful, thanks keithcurtis! I've also been thinking about what I want to achieve and I think it's selfish to look at how I can benefit without first working on something I can share with the community so I'm going to focus my efforts in a different direction.

Cool.  Definitely post back with any API questions.  One thing you might consider doing is sending the output in such a way that it is not visible by default, but your scraper can still pick it up.