Ghosts Don’t Need Social Data

It’s Hallowe’en, and that means we have a very special (and perhaps, slightly creepy) October blog post in store!

Keeping the lights on

As I was going up the stair
I met a man who wasn’t there!
He wasn’t there again today,
Oh how I wish he’d go away!

When I came home last night at three,
The man was waiting there for me
But when I looked around the hall,
I couldn’t see him there at all!
Go away, go away, don’t you come back any more!
Go away, go away, and please don’t slam the door…

Last night I saw upon the stair,
A little man who wasn’t there,
He wasn’t there again today
Oh, how I wish he’d go away…

– William Hughes Mearns,
“The Little Man Who Wasn’t There”, 1899

Let me set the scene: it’s mid-Fall and you’ve gone out for an evening stroll in a part of your neighbourhood that you rarely visit. There are fewer houses about, and you find yourself on a lonely path as fog closes in around you.

Whistling nervously to yourself, you round a corner and, through a parting in the trees and mist, you spot a house at the end of the path. It’s dark, and the only thing that gives the house away is a solitary light gleaming from a top floor window — the attic. No one is around, there’s no sign of activity and the house looks to be in a state of disrepair.

You might be forgiven if your first thought turns towards the most illogical conclusion: this place is haunted.

Those of us who have children (or qualify as “angry old men”) might also be forgiven if they find themselves thinking, “Who turned that light on? Don’t they know electricity costs money? Ghosts don’t need lights…”

The Man Who Wasn’t There…

Tales like this may be spooky and, perhaps, entertaining to some. However, they are just that: tales. Fictional accounts for the thrill-seekers amongst us, told with the understanding that ghosts are not real — there really is no one there.

As with philosophical questions like “what’s the sound of one hand clapping?”, or “if a tree falls in the forest, and no one’s around, does it make a sound?”, one might ask “If there’s nobody home, does anyone need the lights on?”

Unlike philosophical questions, however, this one has a definitive answer — “No.”

At Rock Content, this summarizes our thinking around an update we recently introduced to the way we handle social automation.

Engage is supposed to work as follows: if a live blog has seen no activity from audience members or moderators in the past 3 hours, it will temporarily close. This is the way our app has always worked, and it is very intentional. There’s no reason to keep the stream running during off-hours, especially when it’s not being read or monitored.

This automatic closure ensures optimal performance, and it’s very easy to reverse: simply log into your account, navigate to a closed stream and click “Open” in the text editor space of Content Studio.

There’s one catch to the above, however: this hasn’t been working the way it’s intended when a live blog has an automated social feed. Automated feeds continuously bring content in from various social networks and, until recently, our app has interpreted this content as activity.

The fact is, social feeds bring in a lot of data. Let’s look at a relatively moderate example from Twitter…

In the above image, I’ve searched for #halloween on Twitter. Sure, it’s timely and there’s probably a higher-than-average number of people tweeting with this hashtag. That said, it still doesn’t compare to higher-volume hashtags, especially those focused on global events or perennial subjects. Even so, within the span of a minute, I was able to count 54 tweets — almost a tweet a second. Let’s extrapolate that across an entire day…

(54 tweets/min) x (60 min/hour) x (24 hours/day) = 77,760 tweets/day

Now, let’s look at the hourly activity that we see on our platform throughout the day:

This chart is a heat map that measures the average volume of users active on our platform on any given day of the week, broken down across each hour of the day. The red boxes highlight hours of limited activity, where few users are engaging in a live blog discussion.

Using our #halloween example, let’s assume that the total number of tweets per day is the same for each day of the past week. We can’t make an assumption about the relative volume of tweets, as the hours highlighted probably see a variance in tweet frequency. These hours include peak- and low-usage hours, but let’s assume an even distribution of tweets that represent ⅓ of the day’s output, roughly 25,920 tweets.

This means that, prior to our recent update, if someone had started a live blog with an automated social feed that brought in any reference of #halloween, they would be collecting 25,920 tweets during a period where no one is watching. Not just tweets, however, but associated images, videos, audio files and metadata. The likelihood is that we would be storing gigabytes worth of Twitter data for no one. For a person who isn’t there…

That. Is. Scary.

Turning Off the Lights

Rock Content needed a way to address this problem. Storing gigabytes of unwanted and unnecessary data doesn’t just impact us; it impacts you, as our users. Data storage is costly – it costs money to maintain, and it can affect the overall performance of the application. The more data we need to store, the more it costs us to manage Engage. Likewise, the greater the chance that you will see performance issues and errors in your data. We don’t want that, so we decided to do something about it.

As mentioned, we released an update last week that addresses the above issue. Engage will now enforce the “3-hour-rule” for all live blogs without a scheduled close date, even those with an automated social feed active. If a stream sees 3 hours of no activity on either the front-end or the back-end, then it will close.

However, we recognize that some of our users will see this as a change in behaviour. Many of our customers use automated social feeds on a regular basis. Some may have never held a live blog without setting up an automated feed that brings in content from a 3rd-party site. Because of this, we wanted to ensure that we didn’t release this update without a way to notify our users of when their streams close.

From now on, if you’re a moderator or administrator of a stream, you’ll receive something like the following email in your inbox:

This message will be delivered to moderators or administrators whenever a stream closes due to inactivity. This will ensure that you’re never caught by surprise — you’ll always be aware of when your stream is closed, and we’ve even included a convenient link that will take you directly to the back-end of the affected stream. There, you’ll be able to reopen your stream by clicking on the “Open” button in the text editor, as per the screenshot above.

So don’t worry if you see these unexpected emails show up. Your stream isn’t haunted. Your inbox isn’t possessed. And that “View Stream” button is definitely not cursed. We promise.

Lucan is a Senior Product Manager at Rock Content, the Engage live-blogging platform. He has a background in Big Data and Social Media Analytics and has held product roles at several Toronto-based startups in these industries.