What I learnt from building the Product Quiz
A six-month postmortem
This is Day 7 of the EDOCODE Advent Calendar 2024! The previous article, Iteratee, working with data stream, was written by Cosmia Fu, a engineer of EDOCODE. The next article is on the creation of the Clarigo brand, by Wenqian Ma!
Also, please check out the Wano Advent Calendar by our parent group company!
日本語版はこちらをご覧ください。
Say you're in the market for a new coffee machine, and you've settled on a brand, and you're on their website. You want a fully automatic coffee machine (because you're a bit lazy and don't want to grind the beans yourself), and you don't want something too expensive either. There are 20 different fully automatic machines, so which one do you choose?
Enter the Product Quiz. Just answer two or three questions, and the quiz will automagically assign you your perfect coffee machine!
Or so would a pitch go for the Product Quiz feature I’ve been working on for the last six months, as an engineer working on Gojiberry, a survey insights app on Shopify.
When you're in the midst of it, it's hard not to be caught up in the roller coaster of emotions that goes hand in hand with product development. It's December now, and it feels like a good time to do some reflection.
What happened
Month 0
Sherry, our designer, had the original idea of the product quiz. Essentially, the shop owner would create such a quiz to help (possibly confused) customers pick out products most relevant to them. There are similar quiz apps on the Shopify store, but from what we've seen, their UI is often obtuse and hard to use. Also, they usually charge by the number of quiz responses.
We wanted to build something that was simple to use, and have a better pricing modal, where shop owners only need to pay a percentage of the attributed sales at 3%. The thinking was that shop owners would be happy to pay us if our product led directly to sales.
Month 1
Timmy, our product manager, had found a couple of shops that were currently using Gojiberry for our survey feature, and after a lot of effort pitching our quiz idea, agreed to participate in the early beta.
At that point, the team was really worried about how long development took. Inspired by the company hackathon we did last year, I suggested that we could take two engineers, and build an MVP in two weeks.
This MVP really only consisted of the product quiz on the shop owner's page. We wouldn't build any of the quiz creation UI used by the shop owners. Instead, they would need to provide all details of the quiz in a CSV file.
We'd skip some of our usual dev practices that were slowing us down. So we won't write any tests, and do away with feature stories with acceptance criteria entirely. It was a bold new experiment.
So off we went. Saloni and I worked on building the quiz. We split it up mainly in terms of back end and front end. We also got help from Des, another engineer who’s always a great resource for tricky technical questions. That month was the most fun I had at work!
Learning 1: You can move really fast
With a limited scope and focus, we moved really quickly. Saloni and I basically finished the customer side of the quiz in two weeks, and another week for bug fixes.
I'm in two minds about whether skipping testing in the early stage was a good idea1.
But there is much to be said about the productivity gains (at least for me) to just be working single-mindedly on a simple project. You know exactly how it works. Since you have all the context, bugs are pretty easy to fix as well.
Unfortunately, this pace doesn't last, because once the project gets bigger and takes up a life of its own; when other developers start working on the codebase, there'll be knowledge gaps and you'll need feature stories, tests, and other procedures just to stay sane.
Month 2 - 3
When the second month rolled around, the bad news hit. The shops that had initially agreed to participate in the quiz beta either pulled out or weren't responsive anymore.
After some long and difficult discussions, we put it down to the quiz just being too hard to set up. The hardest bit is the product tagging part — where you need to associate your product with the quiz options. For instance, going back to the coffee machine example, you might ask: "What sort of coffee would you like?", and you'll need to tag all of your coffee machines with tags like "fully automatic" or "semi automatic". Essentially, if you had a lot of products, this process would get tedious — you'll probably need a couple of tags per question per product.
We assumed that without the interface to create the quiz, no shop would set up the quiz. So we got to work building out the quiz creation interface.
Learning 2: You can’t assume what your customers are thinking!
While the shops were initially eager, why did they pull out?
We thought it was because the tagging was hard, but that wasn't the issue at all. (I'll get to this in the next section)
In hindsight, I think the real issue was with the complexity of the CSV itself. We were not only asking about the content and logic of the quiz, but all of the styling too2.
The hurdle was way too high. These shop owners are busy people and they don't have time to spend hours figuring out how to set up our quiz.
The mistake we made was this: We didn't really need to go build the quiz creation interface right away. We made an assumption based on their behaviour. Even if something seems obvious, unless you hear it from your customers, you might be wrong!3
Month 3 - 4
We added two more developers (bringing the total to four people working on the quiz), gave up the scrappy MVP way of working, and went back to feature stories and unit tests.
Eventually, Timmy managed to get two stores on board to try out the quiz. The first, Store A, was a big fashion and apparel store. Store B sold fashion accessories.
We'd scheduled a meeting for each of the stores to hand-hold them and set up the quiz. The quiz creation interface wasn't finished yet at that point (in fact it was missing a lot of the CSS styling and looked rather terrible), but we wanted to move quickly.
This brings us to...
Learning 3: See how customers use your software
Just being in the same (virtual) room while our customer set up the quiz was incredibly valuable. We caught lots of bugs and usability issues that we'd overlook during development. For instance, swiping back on the laptop trackpad would make you lose all quiz progress.
Store A did not plan out the quiz beforehand, and we just used the time to see them play around with the quiz setup interface.
Store B, however, came fully prepared. They knew exactly what questions to ask, and which products corresponded to which quiz options. They'd made an entire flowchart themselves.
It's also interesting that they didn't want to show products individually, but wanted to recommend a set of products. This eventually developed into the collections idea.
But what's more, is the insight that shop owners could figure out how to set up the quiz on their own. (So that wasn't hard!) Shop B just used our software to implement their quiz.
In essence, seeing how customers use your app live is a great way to see what can be improved, and get new ideas.
Month 5 - 6
We finally launched the quiz for Shop A, where we ran an A/B test to see if doing the quiz would boost conversion for those who did the quiz versus those who didn't see it at all.
Learning 4: Fail and try again?
After a month, we had a look at the results, and unfortunately they weren't good... Basically, there was no difference in the conversion rate.
Why is that? The worst case scenario is that no quiz -- no matter how perfectly crafted -- will increase conversion. (This means there's no real point in having a quiz on your store at all, apart from the thrills you get from doing one).
However, it could be that how effective a quiz is will depend heavily on what questions you ask and the products you sell. For example, clothing might just not be well suited. Shoppers usually know what they want to buy, and the styles they're into. They don't need a quiz to tell them that, and they probably won't trust the quiz either.
And then I thought about my own experience of buying a coffee machine. I remember doing a similar product quiz, and in that case, a product quiz could work in areas where you need domain knowledge to choose between different products effectively.
It can be hard psychologically, dealing with the ambiguity of not knowing whether a feature will work out. Sometimes it really comes down to experimentation. A lot of the time, you don't know what makes sense, and you just have to keep learning and keep experimenting.
In conclusion
Hindsight is 20/20, as they say.
I think we did some things very well -- the minimal MVP approach was the right approach. Although, if we were to do it over again, I'd want to get our prototype out sooner, since that's where most of the learning happens.
We spent the bulk of our time and energy on the quiz creation screens. I might be wrong on this, but we could have held back until we got more feedback.
TL;DR: When making a product, move fast, stay focused, see how users use your app, always question your assumptions, and keep learning and experimenting!
At Wano Group, we are hiring! If you are interested, please check out our open positions using the link below:
JOBS | Wano Group
Testing
To me, testing is good when you’re sure of the behaviour, and you want to make sure you don’t break anything. The tests themselves are part of the software you’re writing, and eventually make the code hard to change (since you’d then need to update all the tests too). It’s a trade off between the stability of a system vs the ease of modification of the system. After all, you don’t want to be introducing a whole bunch of bugs with every new change.
On the other hand, if you’re not sure how your system should behave – that is, if you expect to change it very soon – then maybe writing rigorous tests will just slow you down.
In our case though, we had a good idea of what we wanted the app to do, and to my mind, there weren’t many drastic changes we needed to make.
Eventually, I reckon around two months in, the tech debt really started to impact productivity, and then we were sort of forced to go back to writing tests just from the sheer number of bugs that were popping up.
Quiz styling
This was another mistake on our part. We’d assumed that shop owners would know how to set up a beautiful quiz that complements their store aesthetic. Not so.
Later on, our team basically redid the design for Store A’s quiz, since their design (made using our quiz creation workflow) left much to be desired.
What we should have done
Knowing what I do know, we should’ve done this:
Ask the shops to think about what products they’d want to put in the quiz (think about what questions to ask, what products to recommend, etc.)
Book a 1-hour meeting with them
Figure out how they want the quiz set up
Later, fill out the CSV settings ourselves, design a nice quiz, and show them
Essentially, we should be following the familiar startup advice of “doing things that don’t scale.”