By Michael Woloszynowicz

By Michael Woloszynowicz

Saturday, February 19, 2011

Crowdsourcing Your Way to Average

Upon first glance crowdsourcing appears to be a great way to accomplish just about anything. After all, the old saying goes that two heads are better than one, so naturally hundreds of heads make it all the better. After some recent experiences with both CrowdSpring and 99 Designs, I would argue that crowdsourcing is only beneficial in certain circumstances and restrictions.

With the growth and success of sites like Wikipedia, Reddit, and Stack Overflow, it's clear that crowdsourcing can work really well. The key difference however between the aforementioned sites and for-profit sites like CrowdSpring, is that members of the former are driven by the desire to collectively improve the content and offering of the site, while the latter is driven by individual competition in a winner takes all scenario. What I've discovered as a similarity between both types, is that the bulk of the high quality content is submitted by a very small portion of the user base. On collective intelligence sites such as Wikipedia this is still OK as this small percentage represents a large number of users in absolute terms, so they're still able to maintain a relatively high standard of quality. On CrowdSpring or 99 Designs, the end consumer is left to steer the direction of content submissions which becomes an onerous and time consuming chore. This however is not greatest problem with for-profit sites, the greatest problem is that the intelligence is not actually collective. Rather, you get a sort of "1000 monkeys working on a 1000 typewriters" syndrome where content is not gradually refined but instead individually re-invented. I ran my competitions in a public manner so each person could see other submissions and what had scored well, but somehow this did not result in a convergence towards an ideal logo. Instead you had less creative designers submitting lower quality versions of high scoring logos while others started from scratch. Having new styles and ideas come in is great, however very few designers actually took the time to carefully read the project description so you end up spending a great deal of time re-explaining requirements and guiding them towards your desired style. Since your project is not the only one going on, many designers quickly lose interest and move onto another project if their first submission didn't hit the mark.

In addition to the relatively poor outcome of this process there are also several depressing aspects to this sort of product creation. The first is that it doesn't maximize value creation since the combined efforts of so many individuals coupled with the project award value yield a pitiful average hourly rate for all involved. Viewing many of the designer profiles we see that the average win rate is in the 2%-6% range implying that either the process is inherently flawed, or the designer is simply not very good. Either way this does not bode well for the end consumer. The other downside is that while participants in collective intelligence sites are happy contributing content, those in for profit sites seem to resent the process as it drives down rates in their industry. Can we therefore get a good result from a group of disenchanted designers?

This is not to say that for profit sites can't be successful, but I think it's time to rethink the process and the constraints within which they operate. The primary driver for my use of these sites is the challenge I've had in finding a designer whose style reflects my company's taste and direction. In fact we've tried hiring a marketing company to design one of our logos and after months or back and forth we abandoned the process. What I therefore suggest, is that these sites strive to become more of a directory for design talent where designers compete not just for money but also overall standing (e.g. Stack Overflow). What I'd like to do is start by perusing a large set of designer portfolios and choosing those designers whose past work is appealing to me and reflect my companies product and design styles. At that point I invite a small number of them to take part in the competition. They review the project requirements and if they feel the project matches their design skills, each submits a few initial ideas. At each round of submission we eliminate one designer and rate designs based on design quality, originality, and applicability which add to the designers overall rating. By using a score designers are more motivated to provide high quality content as its effects are carried beyond that single competition. By starting with a small pool of designers the chances of winning are much higher, designers are more engaged with the project, and don't have to spread themselves as thinly across a large number of projects. What we are left with is a process that weeds out low quality designers while maintaining a good effort to reward ratio for the higher performers. Further it wastes less of the clients time and allows them to work more closely with the designers to achieve the desired result. While this process has not been tested in the field, I suggest it as an alternate way of thinking about crowdsourcing in the context of commerce. We need to find and cultivate an environment that emphasizes value creation rather than one sided value capture and we can only achieve this by experimenting with different business models. For a product to achieve sustainable success it must make both the content creators and consumers happy, and as it stands I don't think either side is.

In the end we asked for a refund on one site and were left with a satisfactory, albeit not ideal result on the other. Perhaps running experiments several more times I would have a better gauge of the efficacy of these sites but I can't help but think that it's not the panacea I thought it would be. These are my experiences and I'd love to hear how you've fared using such sites and what advice you may have to make them better, so please chime back in the comments.

If you liked this post please follow me on Twitter for more.

No comments:

Post a Comment