our love fest with
gaming, blogged

Blogs

Mobile Hackathon Success!

Saturday we held our first Boston Mobile Hackathon (pat on the back). We had developers showing up with all kinds of experience, exchanging ideas, swapping knowledge, and making connections through an extraordinary networking environment. It was great fun all while being super productive.

The goal of the event was to showcase some brand new mobile dev tools that are not yet publicly available and bring mobile apps to the next level by adding social networks and cloud service hooks. It was encouraging to see the dedication of all the hackers. We will definitely be hosting another Hackathon in the near future.

Participants submitted video demos of their apps or games to the judges and a winner has been determined based on the best use of the "Vixivey" mobile technologies. That winner is Dave Owens (pictured below: middle) from TapWalk who built a massively multi-user real-time continuous game of the geek classic "Rock, Paper, Scissors, Lizard, Spock!" Congrats, Dave!

A big thanks to Kinvey for co-organizing the event, to WorkBar for providing us the perfect location, and to b.good & Hot Tomatoes for keeping our hackers energized by filling bellies with the best grub in town.

Hack on!

You Really Should Let Us Handle That

You're reading this copy because it spilled out of my head, I popped it into a doc and pressed a few buttons that served it up for you in whatever device or medium you have handy.  Your car may be reading it to you as you drive, you may be scanning it on your phone on the metro, or at your desk as you sip your soup.  What's cool is that you no longer wonder or care what happened between my writing it and your uptake.  Someone, or something else is responsible to make sure the bits were posted, a network moved it, your device could find it and give it to you, and someone actually got paid to enable that process.

At our upcoming Hackathon in Boston, we'll be sharing a first glimpse of our Android SDK for mobile app developers who want to add social hooks to their apps and games.  Sure, some Android games can connect to your personal network of friends on Facebook now, but how does that happen?  What the players don't see or likely care about is how much time the app developers need to spend keeping their apps updated to take advantage of the back-end services that the social networks provide.  That's where Viximo shines.

By leveraging this newest SDK, mobile app developers can write their social network integrations once, and we'll take care of the heavy lifting and updates when they're ready to open their games up to the world of users on multiple social networks.  Heck, we'll even take care of native messaging, real time presence detection, recommendations and tracking requirements for those platforms too.  We've been doing it on the web for years, and now we're going mobile.  Learn more at the Hackathon and watch for detailed resources posted to the site coming soon.

  ...

Know Thy Audience

Filed In:

The job of the Vice President of the United States is one of the most vaguely defined in our political system, but it consists of two absolute necessities:

         1. Be alive in case the president dies

         2. Help the president get elected (or re-elected)

Last week Vice President Joe Biden failed miserably in his latter responsibility when he tried to rally support for Barak Obama by telling attendees of a San Francisco rally that "the Giants are going to the Superbowl!."


As we know now, the Giants will in fact be going to the Superbowl, but to the chagrin of San Francisco fans everywhere and as anyone who's ever watched an NFL game knows; the Giants play in New York (New Jersey to be precise). Old Amtrak Joe most likely confused this team with their baseball equivalent (who do in fact play in San Fran), but nonetheless the gaffe is sure to stay with him as part of his legacy as Veep.

Not just a comical punchline destined for SNL's cold open, Biden's mistake illustrates a crucial core competency to successfully bring a product to market: Know thy Audience. I don't need a focus group to tell me that people will be more receptive to the product I am offering them if they think I can understand their needs.

History is littered with examples of companies large and small that dove into new markets without a full understanding of their audience:

·   Disney almost bankrupted their new park in Paris because they didn't serve wine, a staple of the European meal.

·   Coca-Cola had to recall their 2 liter bottles from Spain because local refrigerators couldn't fit them.

·   An American airline promoted that they offered "rendezvous lounges" in Brazilian airports without realizing that in Portuguese, "rendezvous" refers to a place to have sex.

Mistakes like this can undo months of planning, be incredibly expensive and worse of all- indelibly harm a brand's reputation among a demographic. At Viximo we are constantly expanding into new markets, so it is crucial that we take the time to understand the habits of our new audience. Not all games, branding, or marketing tactics will work equally well across our portfolio. That's why we take the time to work with our publishing partners to learn about their users, examine successful games already in the area, and research behavioral and social tendencies of the region, to decide which of our various games and promotions will work best to maximize our audience's enjoyment. Doing this kind of legwork before launching in a new market ensures that we are never encouraging our audience to do something like root for the other team in a playoff game. ...

Automated Partner Monitoring

Filed In:

Viximo thrives on reliable services: internally and externally. Downtime, even if caused by a 3rd-party, means fewer customers and real costs - not just for us, but for all of our partners. It's for that very reason we've invested a little blood, sweat, and tears into automated monitoring for the partner systems that we rely on.

Back in the old days...

It's hard to believe it was a little over a year ago when we first automated monitoring of the 3rd-party games that run on our platform. At the time, we were still fairly inefficient at detecting outages. If a partner happened to detect an outage it was typically long after it had already started, leaving many users with a cryptic error page. If users created support tickets about the outage, the lack of a dedicated customer support engineer meant we were often backlogged several days. If no employees at Viximo or its partners played the game on any given day, it was possible no one would realize it was down.

The nature of our 3rd-party game integrations makes detecting outages a bit more difficult. Like many apps on Facebook, games are integrated via an iframe on the page. Simple failures such as non-200 status codes are easy to detect, but games can fail in much more subtle ways:

  • Failures isolated to certain geographic zones (US, Germany, Spain, etc.)
  • Issues encountered only within Flash
  • Javascript errors or failure to load certain files on the page
  • Intermittent failures due to capacity issues

Without direct insight into the actual application and the servers it's running on, these problems are only magnified.

Getting the job done right...

The tools we now use are insufficient by themselves, but together they can detect the majority of the outages that our partners experience by tackling the problem from various angles.

Monitis is a hosted monitoring solution that we use primarily for its uptime service. It offers geographic locations that map well to Viximo's social network integrations and provides a suite of advanced notifications / callback hooks.

Airbrake collects errors generated within our application. This also allows us to track javascript errors generated on the browser, particularly within partner pages.

Zendesk is a customer support management application we use to track data about support issues. Zendesk has a great API and management interface that makes it easy to analyze and categorize the various issues per game.

Nagios is a system monitoring application that is used, in this case, to monitor application and user behavior through Updawg. Viximo also uses Nagios for other internal systems monitoring.

Using the above tools, we've defined a series of triggers and notification channels that can quickly alert us to issues so that they can be resolved promptly. The first set of triggers below will automatically mark a game as down and place it in maintenance mode. To prevent blips and false alarms from taking apps down too often, these triggers must fail a certain number of times consecutively from a subset of our 5 geographic locations around the world.

  • HTTP status code - Any 2xx status code is considered success; all other codes are considered a failure. For the most part, this trigger catches most outages since the web server will typically return a 500 status code when down.
  • Content matching - Any response that does not include a particular set of content specific to that game is considered a failure. This catches instances where the game is returning a 2xx HTTP status code even though it failed to process.
  • Timeout - If the url takes more than 10 seconds to process, the game is marked as down. Disabling apps as a result of a timeout can help ensure that the game-playing experience is tolerable for active players. This way other players can get funneled to other games while the performance gets investigated.

The remaining triggers use the available notification channels to alert folks at Viximo when they're failing. This gives us the ability to manually test the game and validate that it is in fact down prior to actually marking it as such in our system.

  • User activity - Visits, transactions, and revenue are compared to historical averages. If any value deviates too far from the average, an alert is generated.
  • Customer support rate - If the rate of customer support issues for a game deviates too far from the historical average, an alert is generated. While this data is available for automated monitoring, this is still a manual process that our customer support engineer performs.
  • Error rate - If the total count for a particular type of error exceeds a certain predetermined threshold, an alert is generated. Again, while this data is available for automated monitoring, this is still a manual process the our engineers must perform.

Communication and recovery...

Once one of the above triggers is activated, a variety of communication channels are available to alert a targeted set of people at Viximo. The type of channel used depends on the severity of the trigger. They include:

  • SMS - This isused for the most reliable triggers, such as those that automatically mark a game as down. Typically the integration manager is notified of these events.
  • E-mail - All triggers that are activated will send an e-mail to a mailing list at Viximo that includes folks from the integration, engineering, and product teams. This ensures that those with knowledge of the game can address any issues should they come up.

In addition to the above two channels, there are also additional communication channels for users on the Viximo network:

  • Maintenance page - When a game is marked as down within the Viximo system, a maintenance page is automatically displayed in place of the game. This includes a friendly error message and points users to other games that can be played.
  • Announcement - Administrators have the ability to create announcement popups that will be displayed to any user attempting to play a Viximo game. This also allows us to quickly alert users when there are problems in a game that we're investigating.

Once a game outage has been validated, the partner is typically contacted via e-mail and phone so that they are aware of the issue. An integration engineer is made available in cases where logs and error data are needed to help explain what's occurring.

Moving forward...

Like most software development, monitoring is an ongoing process that can be constantly tweaked and improved to more accurately and quickly detect and resolve outages. There are still some triggers that we could automate, such as the detection of increases in customer support requests and application errors. As well, we could begin to take advantage of Monitis's full-page website monitoring tools that allow every link on the game's page to be validated instead of just the main page itself.

While we strive to make content available to our users as reliable as possible, there are always going to be unexpected outages. The best thing we can do is to make that experience as painless as possible for both our users and our partners. By having automated monitoring in place for our partners, this has put us on the right path towards that goal. ...

cloud server ! = cloud server

Filed In:

One of the more interesting facets of my work at Viximo is tracking EC2 cluster performance. Below you’ll see a graph of data collected and displayed through New Relic.

The graph shows throughput and average response time for our cluster over a three hour period. The vertical bar in the center represents a feature hotfix that most certainly would not affect performance. So why would response time drop so dramatically from 33ms to 27ms, a (33-27) / 33 = 18% difference?

Spinning up instances in EC2 is cheap, and restarting passenger results in starvation while rails initializes and is then forked into workers. Instead of restarting apache we spin up a new batch of servers, wait til they’re settled then re-point the load balancer. The graph is illustrating the differences in cloud hardware/instances.

The AWS zone we're in appears to have three different classes of hardware for small/medium instances, fast, slow and screwy as illustrated above. Slow as in 40% slower than the fast instance. Screwy as in what’s up with that reported cpu? (Known ubuntu kernel bug). At first we’d thought the instance performance differences could be explained by something obvious like multi-tenancy, however after lengthy capacity testing we’d found that the data in /proc/cpuinfo mattered significantly more than other factors such as multi-tenancy. The variation between a fast and slow server could be 25ms vs 40ms average response times. Inside the fast hardware class, multi-tenancy and other factors explain the observed range of 23-28ms average response times.

Back to the inital graph, the average response time for the 6 instances is (25.2 + 24.3 + 41.4 + 25.1 + 26.5 + 29.9) / 6 = 28.7. Now change two of the fast instances into slow instances (eg 24.3 => 41.1), (25.2 + 41.1 + 41.4 + 40.5 + 26.5 + 29.9) / 6 = 34.1. Boom, we’ve got a (34.1 - 28.7) / 34.1 = 16% difference.

So what does this mean for Viximo’s day to day operations? Most of the time the differences are ignored. Occasionally we find it worth the effort to weed out slow servers for long running rarely modified services. Sometimes it does matter such as a Ruby app tier where cpu is the limiting factor and the cpu differences can either degrade performance or conversely cost us money. For those we must capacity test each class of hardware then factor in observed ratios for cluster sizing.
...

Behavior By Gender: Diversity Series

To follow up on my last post, I decided to take a look at how men and women use social notifications on our platform. For the uninitiated, notifications come in three flavors:

     Invites, which are sent from one user to another, encouraging new users to come play the game
     Messages, also sent from one user to another, often used for sending gifts or other specific items
     Activities, which are posted to a users’ wall or other social feed, highlighting an achievement, mission accomplished, level-up, etc.

Looking at behavior over the past 6 months, activities by far are the most utilized form of social sharing on the Viximo platform. Nearly twice as many people have posted an activity than have sent an invite. That doesn’t mean that activities are more efficient for viral growth; the conversion rates from invites and messages are higher than from activities. In other words, users are more likely to click on a private message from a friend, rather than an activity post that is likely to get lost in their social feed.

What about gender differences? On average, men on our platform are 10% more likely to post activities to their social feeds. Women, on the other hand, are nearly 20% more likely than males to send invites. They also tend to invite more friends (22 on average, vs. 18 for men). Both genders are equally likely to send messages, but women send nearly 50% more messages per user.

Is it any surprise that guys like to blast their entire friend group with posts about their accomplishments within social games, while gals prefer one-on-one interactions? Maybe not. But considering the long-held stereotype that social games are made for bored housewives, I wasn’t expecting men to be big social sharers when it comes to game activity (at least not publicly).

Notifications are an important component of social game play (they put the “social” in social gaming). Obviously, they are key to growing your user base, but can also contribute to retention. It can be a challenge for both game developers and social networks to make the most of viral channels. Knowing how men and women use different types of notifications can be a key part of the optimization process. After all, viral channels are free, and who doesn’t want to reduce their marketing costs?

Anyone else out there surprised by these results? ...

Happy Holidays!

Isn't our tree lovely? Happy Holidays everyone!

 

Warm wishes,

How We Build

Filed In:

I'm excited to announce that in the weeks and months to come, we're going to begin sharing more about our technology, engineering, and operations.  We're eager to share how we've designed and built our service and some of the many lessons we've learned along the way.  But first, to help orient those discussions, I want to give you a glimpse inside just how we build our products.

Application Engineering

We call our core engineering team Application Engineering.  This team is primarily focused on development of any user experience components, our integration container, and all the RESTful services supporting those.  Essentially, anything outward facing that our partners or users may see is developed by this team.

Requirements for the App Engineering team are managed by our product management staff which includes both product managers and partner account managers.

Because these features often have deadlines, or at least timelines that need to be communicated publicly, this team uses Scrum as it's primary process.  Sprints are typically 2-3 weeks in length with a set of target stories and a planned release date.  We find that Scrum also allows for enough rigor to implement a solid testing period before the release goes live.  The burndown chart is shared throughout the company and the team holds daily standups including stakeholders to track progress. 

Of course, throughout the sprint we regularly perform update releases, sometimes several in a single day.  So we're not stricly bound by the sprint schedule, but it provides a nice cadence to feature development and helps manage expectations with our product team and our partners.

Infrastructure Engineering

Over time we discovered a different category of work that we call Infrastructure Engineering.  These tasks often have more to do with systems than features, things that affect performance, scalability, deployability, or monitorability.  They are not overtly outward facing.  This team includes what most think of as devops, but also includes architecture and application work as well.

Requirements for the Infrastructure Engineering team are primarily managed by our technology team which takes input from engineering and operations.

Because these features don't typically have rigid deadlines, but it is important to manage and prioritize this work, the team uses Kanban as it's primary process.  Our Kanban board is shared with everyone and the team meets weekly to talk about approaches, review impediments, and raise new issues.

Work from the Infrastructure Engineering team rolls into production all the time, but some architectural changes are obviously coupled to App Engineering sprint releases to keep things in sync.

Extreme Flexibility

The trick to all of this is that we're still a completely flat organization and these teams are extremely fluid.  Typically the App Engineering team has about 8 engineers and the Infrastructure Engineering team has about 4 engineers, but this changes all the time.  As a new rush of application oriented work comes through, we'll shift resources towards App Engineering.  As a large amount of infrastructure work is required we'll shift some staff in that direction.

This also means that engineers have the opportunity to drift across the product stack, getting great full stack exposure and changing up their work on a regular basis.  Several of our engineers very intentionally alternate back and forth between application and infrastructure work on a regular basis.

This organization has allowed us to deliver dozens and dozens of releases on-time and on-plan, while continuously evolving our architecture to meet the growing needs of our partners and customers.

Hopefully this will help orient some of what we'll share in the months to come, and if you're interested in joining our team, we're always looking for more smart folks.  Please email us at jobs@viximo.com if you think you can help. ...

Viximo launches Backyard Monsters on Yahoo!

Filed In:

POPULAR SOCIAL GAME NOW ON YAHOO! GAMES AND YAHOO! MESSENGER

San Francisco – Nov. 9th, 2011 – Leading social games and apps platform Viximo has launched wildly successful resource management strategy game Backyard Monsters, developed by Kixeye, on Yahoo!.  The launch of the game on Yahoo! Games and Yahoo! Messenger marks a new relationship between Viximo and Yahoo!, which has 700 million worldwide unique visitors per month.

Backyard Monsters is one of the most popular games online and currently has more than 4.3 million active users on Facebook.  The game allows players to design and build towns for their monsters and challenges them to produce three types of essential resources, a town hall, facilities to make a monster army and storage buildings to hold building materials.  If they design their town correctly, players will be able to protect their buildings from invading monsters.  Backyard Monsters joins the town-builder simulation game Township, developed by Playrix, which was also recently launched by Viximo on the Yahoo! Games page and the Yahoo! Messenger client.

“Viximo’s developer partners have a unique opportunity to reach new audiences and be ahead of the curve by diversifying their distribution strategy, and our social network partners know they can count on us to be matched with the most immersive and highest monetizing social game titles,” said Dale Strang, CEO, Viximo. “Backyard Monsters is a top social game that will be a great fit for the Yahoo! audience.”

“Viximo’s platform presents a new and effective way for networks with a strong social context to integrate some of the best social gaming content in a way that will drive user adoption and monetization while providing a fun and engaging experience to our users,” said Lee Parry, director, Messenger & Mobile Communications, Yahoo!.  “We’re looking forward to adding top-quality titles such as Backyard Monsters to our millions of users on Yahoo!”

Viximo’s platform distributes leading social games to an international and fast growing portfolio of partners, matching the best social games to the world’s top social networks.  Optimizing for gameplay, virality, discovery and monetization, Viximo enables developers to quickly and easily diversify and globalize their distribution strategy. 

Viximo’s global network features more than 20 social networking sites around the world, including Tuenti, VZnet, IMVU, Gaia Online, MyYearbook, Quepasa, and Orkut, Brazil’s largest social network. Viximo has a global reach of more than 170 million. ...

Social Games Engender Diversity

Now that social games have evolved past simple farming games to a full-fledged multibillion dollar industry, does anyone still believe that the typical social gamer is a 43-year-old woman? While most of the initial games on Facebook appealed to this demographic, social game content has grown in both volume and variety. Sure, there are still a lot of clones out there - look at the sheer number of farming games - but new genres and game mechanics have emerged that pull in a much wider audience.

To demonstrate this, I took a look at three of the most popular games across the Viximo network, focusing on gender.

GAME 1 - Big Business. This graph is probably what we would typically associate with a social game. Females account for about 60% of the users, views, and revenue for the game across our network:

 

GAME 2 - Ravenwood Fair. This game, deployed on similar sites as Big Business, has a slightly different pattern. Males make up half of the audience, but are much less engaged and certainly don’t monetize as well as females:


GAME 3 - Backyard Monsters. On the other end of the spectrum, this game blows the whole “housewives are the only ones playing social games” theory completely out of the water.


As these data show, social games aren't just for women in their 30's (despite what your core gamer friends might have you believe). As with traditional games, social games can appeal to many different demographic groups by providing carefully tailored content and functionality. Each segment of users has different - and potentially advantageous - usage habits and spending patterns. This is where Viximo comes in.

As a social game distributor, we use a game’s demographic data (among other things) to inform our approach to advertising, cross-promotion, community messaging, sales and incentives, and more. Making a great game is a big part of success, but getting it into the hands of the right users is even more important. ...