 |
Jason Cohen founded Smart Bear Software on a bet. The bet paid off. Profitable and cash-flow positive since Q4 ‘03, Smart Bear was recently acquired by the software testing firm, Automated QA. During a recent conversation about agile practices, Jason mentioned that Smart Bear was employing agile practices in their marketing strategy. I asked if I could interview him for GeekAustin. |
Lynn Bender: Jason, it seems like only a few years back when GeekAustin saw the first Smart Bear job posting.. Since then, you’ve gone on to win the Jolt award for Code Collaborator. I believe, however, that Code Collaborator wasn’t your first product. Could you tell me about your suite of tools and the market they serve?
Jason Cohen: Yeah, it was just a few years back when GeekAustin ran our first posting. By the way that’s also where I found employee #1!
Code Historian was our first product — a version control system data-mining tool. Version control is used primarily to keep developers from stepping on each others’ toes, but there’s a lot of useful data in there that’s generally hard to get at. We had everything from a “time-line diff” where you could get a diff of a file between any two points in time with one click to a system that transferred version control data to an RDBMS for real reporting.
Code review wasn’t in the original game plan. I stumbled into this hole in the developer tools market because people were abusing Code Historian in order to do reviews. By listening to my customers I made my way into code review.
Sounds agile already doesn’t it? Morphing yourself based stakeholder input rather than sticking to some pre-determined long-term master plan. If I hadn’t done that, we would certainly not have the same success.
Bender: When we last discussed agile methodology, you mentioned that Smart Bear employs agile methods in business and marketing. Could you elaborate on this?
Cohen: The traditional business plan looks out 2-4 years. Marketing — especially in print and events — takes months to plan, expects targets to be met months in the future, and campaigns often run 1-2 years.
These time-lines are as far-sighted and pre-planned as traditional waterfall software development, and at Smart Bear we don’t have them. Our marketing efforts can last 1-2 months, and can be set up in less than a month. We measure the response and adjust accordingly.
Bender: Traditionally, it seems that marketing has been driven by that elusive quality called inspiration. Once found, this is usually followed by pre-launch preparation, and then a massive launch. This is a heavyweight approach. Are there ways to insert agile practices into these steps — or do you simply have to discard this approach entirely?
Cohen: It’s a tricky answer because some things in marketing are beyond your control.
For example, print ads have a 2-3 month lead-time, and often you have to sign up for 6-12 months. Tradeshows can have a 12-month lead-time to get involved and 6 months lead-time for preparation. You can’t change these timelines, so to some extent they exert non-agile control over your efforts. For example, if the new print ad talks about features in v4.0, it’s going to come out 3 months from now and you’d better have released those features by then!
However the majority of our marketing efforts are, in fact, very different from the usual approach you described. We don’t talk about artsy-fuzzy things like ‘inspiration, we start early, and we fail-fast. We throw out the traditional model completely except for these issues of timeline.
Bender: Your book on code review is prominently advertised on the Smart Bear website. How does this fit in to your agile marketing strategy?
Cohen: We give away something of real value — a book about code review. It’s a physical book and and we print and ship it for free. There are sample chapters on-line so you can see that it’s full of genuinely useful information, not a sales pitch. We ask where you heard about us and people tell us. We know they tell the truth because if they came in from a web page we save the referring URL and we can see that the text they type in does match the referring site, so we assume it’s accurate for the other marketing efforts too.
We know books lead to trials because in our live demos the potential customer has always gotten the book. Over 50% of people who get a live demo end up purchasing, so there’s a direct and substantial correlation between getting books and purchasing.
We use the book to determine which marketing efforts work. Sure, it costs money to print and ship books, but what’s the value of not only promoting your product and earning trust in your company, but simultaneously measuring the efficacy of every marketing campaign?
Bender: Programmers have a suite of pre-built tools to drive the development process. Can you tell me a bit about how you manage test-driven marketing? I am guessing that there is nothing like JUnit for marketing.
Cohen: Ha, yeah there’s not.
The idea of test-driven development is to set up the conditions for success, then write code until the conditions are met. It also implies that you continue to measure continuously in the future to make sure you don’t regress.
We approach our marketing the same way. It’s only a question of time and money to get a print ad in 7 magazines, to get a booth at 20 tradeshows, and to spend $10k/month on Google ads. But you don’t have infinite time or money, and you know half those marketing efforts are a complete waste. So you have to measure which ones work and only spend money there.
So first set up the test cases. Maybe it’s “We get an average of 100 leads/month from this ad.” Or “We get 5 sales from that tradeshow.” Then you try it, and measure it (measurement is the hard part). Then it either passes or fails. If passes, we might spend more money but we’ll definitely stay with it. If it fails, we’ve spent a known amount of money, that’s OK.
Then there’s the “continuous testing” part. We’ve found that in marketing it’s often true that an ad can work in a magazine for 2 years and then stop being effective. So there’s the regression bug testing aspect as well.
We just use a single table in a database for book orders. We have a bunch of regex’s that look through user-supplied “how did you hear” (it’s fill-in-the-blank to prevent people from selecting an item at random) and mapping it into a set of 30+ fixed categories. Then we just “group by” that and count the books, usually grouping by date etc.. We use MS Access to generate a few reports.
One of the neat reports is “ROI.” Here we have a little table (updated by hand in Access) with how much money we’ve spent in each campaign, then we link that to the book orders. So it’s “Cost per book” on the advertising side; add in the cost to print and ship times the number of books and you have your total expense.
Bender: Can you tell me a bit more about the test metrics?
Cohen: It’s often difficult to link a sale to a marketing campaign, which is ultimately what you want to know. That is, if I spend X dollars on campaign C, I eventually get Y dollars in revenue. Most people are happy if Y > X, even if Y isn’t much bigger, even with overhead. We’re like that.
But tying the X dollars to the Y dollars is a lot of steps. When they got from your ad / booth / mailer / phone call to your web site / sales rep, did you record that fact? When others in the same group try out your software, do you link that to the original guy who found you? When you get the purchase order from accounts payable, can you link that as well?
Often even the first step is hard. Print advertising is notoriously difficult. You can try a fancy URL, but techie people know that’s unnecessary. Print-ad sales reps tell you you’re buying a brand, but that’s not my experience.
Bender: Putting the stakeholders first is considered to be one of the core agile values. Can you tell me a bit how this works in marketing / business development?
Cohen: It’s tautological to say that marketing should be driven by customers’ desires, behaviors, pains, etc., but so often it just isn’t.
Take IBM WebSphere (http://www-306.ibm.com/software/websphere/). It’s almost impossible to get any real information from the site. Tons of links to nebulous places, empty text like “Extend access to business processes, applications and information to anyone anywhere.” Was this site designed to serve a potential customer, or just a hierarchical dumping ground for a mountain of information?
Contrast with Safari (http://www.apple.com/safari/). In 5 seconds I can see what it is. I can download with one click, or I can get screenshots and brief, tantalizing descriptions of what it does. Isn’t that what you want from a software web page 90% of the time? This is designed with the curious, time-strapped user in mind.
We’re always in our customers’ heads. Everything we make — print ads, website, product screens — we ask ourselves questions like: What is the use-case? Is this really what I want to see, or is this just what we’re trying to push down their throats? Is this serving a specific purpose or is it fluff? Are we communicating something useful with every phrase or are we speaking in generalities? Are we really solving a specific problem for the customer?
If you don’t ask these questions, you’re not treating your potential customer (the ultimate stakeholder) as the most important thing.
Bender: Do you ever think we will hear marketing people talk about SCRUM and sprints?
Cohen: I doubt it. Although it would be nice to get iterations, 2-week deadlines usually not important in fact.
The “meetings” part of scrum is to get everyone the same page, but in marketing a lot of times either the projects don’t intersect at all (tradeshow planning versus print ads; once the copy and art is done there’s nothing else to interact on) or everyone works together almost continuously anyway, checking things out many times per day — sometimes even like “pair programming. The “release SOMETHING INTERESTING regularly” is probably a useful technique.
Bender: Will you be in town for the Get Agile with Geek Austin party this month?
Cohen: You bet! GeekAustin parties are always a blast.
Bender: I’ll see you then. Jason, thanks for taking the time.
I am more fortunate than many. I only spend about 20 mins a day on Mopac. Thanks to IT Conversations mp3s, Mopac has become my IT classroom. Leading up to this month’s GeekAustin Agile Happy Hour, I decided to fish around for agile podcasts and conference proceedings. Here are two of my finds:
Agile Toolkit Podcast. There are a lot of great interviews here — enough to keep you busy for a few months. In particular:
Team Agile Interviews. Includes interviews with Kent Beck and Johanna Rothman.
With all these resources, we’ll all be scrum masters in no time. Just don’t forget to stop for gas.
-Linear
