Hi! Looks like you're in . Would you like to go to our Canadian store?

  • Evaluating Roast

    First off, I’m super bad at blogging on time/when things happen. This topic is relevant now, but based off of an experience I had a few months ago. In June of this year I was asked to participate in the first ever World Coffee Roasting challenge in Vienna, Austria. This event was a trial for the World Coffee Roasting Championship, which will start officially next year at the SCAE World of Coffee event in Nice, France. The challenge is a “demonstration of a competitive event designed to feature the talent and skill of the dedicated men and women involved in the artisanal craft of roasting specialty coffee.” I participated in the friendly challenge with six other folks (all men…) from Guatemala, Australia, The Netherlands, Norway, South Africa and Taiwan. Here’s a video that pretty much explains the competition, which is more entertaining than reading whatever I would write:

    I hope that this trial run gave the organizers some good feedback about how to make it a success next year. The logistics of running a roasting competition are a little ridiculous, taking into account the equipment necessary, and the amount of time each competitor needs to familiarize themselves with the equipment, coffees, and the actual roasting and resting. It’s also not exactly a spectator coffee sport, like the Barista competition or the Brewer’s Cup, which include presentations, talking and numerous small tasks that make it worth watching for some (while I do think these competitions are good for the industry, they bore me pretty easily as well). I know roasting is an interesting topic for people, particularly customers of ours who aren’t yet very familiar with the process, but I never quite get it when people ask if they can come in and watch roasting; it’s boring to watch and most likely the person roasting isn’t going to be able to give enough additional attention to explain what’s going on, particularly when there’s a full day of production happening.

    Aside from meeting some other awesome roasters and having a great time doing the challenge, the event put some more gas on the fire regarding thoughts I’d already had about roasting and posed questions I’d love to spark further dialogue.

    Here are some thoughts:

    • How do we evaluate roast across different roasting styles/cultures/ideas of what is a “good” roast?
    • How much are people able to really evaluate roast separately from the evaluation of the coffee itself?
    • What kinds of prerequisites are needed or “tests” do we give judges to make sure they can accurately do that?


    We were given three different coffees to choose from, all generously provided by Café Imports. A Colombia, Mexico and a Brazil. All participants graded and evaluated each coffee to decide which coffee they would use to compete. The kind of obvious choice was the Colombia: high density, few defects and a pretty sparkling acidity. There’s the most room for error in this coffee because you’re not depending so much on capturing subtle nuanced flavors with a roaster you’re mostly unfamiliar with, and there’s a decent amount of acidity to fall back on and boost the overall cup should you take the roast a bit too far after first crack or allow too much drying time or development before first crack. In the spirit of challenge, I chose the Mexico, which had a really nice subtle floral quality and some distinct strawberry notes as well—easily a delicious coffee but less easy to nail in a few tries on equipment you’re not too familiar with (a challenge for all of us, the roasters provided were 6 kilo Giesen production roasters, which none of us used on a regular basis!).

    Me. <-- Here I am, lone user of Fahrenheit. 

    In the final cupping, two of us roasted the Mexico, four the Colombia and one person roasted Brazil. These were all cupped blindly together. The cupping was pretty interesting, as it seemed like there were a lot of different approaches, some very light and in my opinion, on the verge of underdevelopment, some with pretty obvious roast characteristics. While I do believe cupping blindly is the best way to evaluate coffees, it seemed that in this case it might have been good to reveal the coffees at least. The three judges cupped sample roasts of all the coffees, before cupping our production roasts, and I hope that they would be able to figure out what the coffees were but who knows.

    The differences in the roasts were really significant to me. The one that stuck out the most was the “Scandinavian” style approach to roasting, which is very, very, very light. Now, Dogwood definitely roasts light, but we’re quite a few notches deeper than most of these style roasts I’ve had, which are at times even lighter than our “sample” roasts. As a parting gift, Simo of Oslo based Solberg & Hansen, well-respected roaster that’s been around forever, gave me two coffees which I drank on my vacation in Berlin over the next few days. While I could tell these were great coffees, the roast was so light it really didn’t make a good Aeropress or drinking coffee, in my opinion. I’ll just say it: the über-light “Scandinavian” style roast is totally not my thing. Sure, it shows the coffee naked and may be good for evaluating the quality of the coffee, but there is a lot of depth and sweetness that I think gets left out with this light of a roast, and the wheaty, stark acidity overpowers a lot of the delicious potential of the coffee. In Vienna, away from the competition, I had a similar experience at a few of the specialty shops I checked out.

    In our discussions about roast styles, Simo commented on the general “American” style of roasting which is for the most part really dark and oily and gross. For the most part this is true I suppose, if you take the mass of coffee roasting and consumption into account. In the specialty sector, I think most roasters concentrate on a well-developed light roast, but stop before any carbon roast characteristics enter into the equation. This is how I’d categorize Dogwood’s roasting style, “well-developed light roasts.”

    So, for a competition like this that includes different production roasting styles from around the world, what kinds of attributes and levels are the “standards” for judging this type of competition? Below is the score sheet used for the competition, which I believe is based off the Brewer’s Cup Compulsory Service score sheet.

     Roasting Challenge score sheet

    So, this is pretty similar to the green coffee score sheet, which does make sense in some ways. When we’re evaluating the roast of a coffee, we are looking at the acidity, the body, flavor etc etc., but we’re looking at it in relation to that specific coffee that we hopefully already know a lot about. So the question about acidity really is, how did you take advantage of the acidity that was present in the coffee, how did you manipulate it to make it taste great? If the acidity wasn’t the particular strong point of this coffee but it has great flavor attributes or body, how did you make those prominent in the cup and make this coffee taste as great as it can?

    On this sheet there’s a place for “defect” as well. I’ll go ahead and assume that this means roast defect. How should this be scored, and what is a considered a defect? Some are visible defects, like scorching or tipping, but what about the ones that aren’t necessarily visible? I’d say that over development is a defect, but what means over development to me, might not mean the same thing to another competitor from another roasting “school of thought.” Also, roast defects are likely something that would affect the whole roast, so multiplying by cups isn’t necessarily the best option.

    The judges were overall not very calibrated, and had also been drinking espresso all day, for whatever that’s worth. I got scored an 87ish something from one judge and a 74 from another. Hm. They apparently had all agreed on scores for the sample roasts, so they felt the same way about the coffees in their naked state, but that’s one interpretation of the coffee. Unless we were all doing sample roasts, which is totally not the point of the competition, the sample roast provided in isolation was maybe not the best way to base all the judging.

    [Side note: These are all just observations and thoughts I’ve developed based on this experience, I hope that what may be interpreted as criticisms are understood to be general thoughts and prompt discussion on roast evaluation in general, and hopefully help with further development of the competition.]

    At what point do you “know” a coffee well enough in its current state (age, storage etc.) to be able to judge roast? It may benefit the overall competition if the coffee could still be a surprise to the roasters, but that the judges were given more time to learn the coffees well; maybe roast it themselves, cup it several times and play with profiles before the competition.

    At Dogwood we cup our production roasts every day, and as the coffee ages we revisit those profiles, playing around with the coffee on the sample roaster and in our production roaster to see what kinds of flavors we can pull out, what’s available in the coffee at a certain point in time. Below I’ve pasted a recently revised version of our profile sheet, the nifty bit of paper that Eddie uses to record roasts. Originally we had the green coffee cupping form at the bottom, but soon realized that scoring with something designed to evaluate coffee outside of roast (as much as possible) every day wasn’t really helpful to our evaluation of the roast. We were mostly just taking notes and not scoring.


    With this sheet we’re not looking at a numerical value, we’re looking at how we did with these different aspects of the coffee based on what we know they should be. So let’s say we do a roast of the Nereo Ramirez Costa Rica, and we cup it and find that the acidity was much more prominent, complex and delicious than the last roast or what we usually see from it, we’d make note of that and take a look at the roasts and see what might have improved that: time of first crack? Rate of temperature increase after first crack? The weather outside? Eddie’s outfit choice that day? Etc. The same goes if something was less than it should be. This type of evaluation gives us a more comparative value rather than an overall numeric value from a form that isn’t necessarily even relevant. The “defect disguise” category is generally for coffees that are aging. It sounds a little harsher than it is, because we generally try not to source defective coffees! Everyone has a coffee once in a while a little longer than they should, and if it’s not at the point where we need to just kill it, how are we doing with minimizing that “loss of organic matter” taste in the cup? Or, for instance if we’re roasting a Rwanda or Burundi, we may want to keep track of how many potato cups we get throughout the time we have the coffee, which may be helpful information for future purchasing.

    Back to the cross-cultural roast evaluation topic. There are similarities here between roast evaluation and green coffee evaluation and the conflict of taste and style. Great coffees can be widely understood to be great coffees, but there’s still the element of taste and style preference. The difference here is that people who evaluate coffee hopefully cup enough that they can understand what a good wet-hulled Sumatran coffee tastes like, or a Brazil, or a Kenya or a natural… even if they hate it. This is done internally at roasting companies that have various roast styles. People can probably understand what a “good” French Roast is, even if they think it’s awful. if it’s part of the quality control they need to do at their particular roasting company; how does that work in this type of competition, though? Do the judges approach it as “this is a very good Scandinavian style roast” or “this is a very good dark roast”? A dark roast doesn’t necessarily show the skills of the roaster person to maximize the flavors and strengths of a particular coffee, but if the challenge was to cover up a defect or something, then it would.

    I’d like to see the World Roasting Competition evolve into some very specific skills roasters deal with every day. It would be great if all participants had to roast three very different coffees (different origins, densities etc.) and maybe something with some sort of problem such as super high moisture content or something baggy. The scoring would be based on how well you could apply your roasting skills to each specific coffee. You wouldn’t roast a high grown, dense Kenya the same way you’d roast a softer Brazil coffee, for example, so how DO you approach those coffees? How do you approach them differently and make each one cup out amazingly as a production roast in their own way?

    This blog has posed a lot of questions and hopefully material for discussion. Roasting is really fun, and I’m so glad that Trish asked me to participate in this trial run. I’m excited to see where this competition goes, and I hope that the United States will eventually have an in country competition and send someone to the Championship. I also hope there are more women next time!

    Send me any thoughts you might have on this topic! Hate mail also accepted.
    Director of Coffee, Dogwood Coffee Co.


    Thank you also to Trish Rothgeb and Filip Akerblom for asking me to participate and putting the competition together, Giesen for donating actual production size roasters (though still small for me!), Cropster, Sinar Technology, Probat, Loring, Curtis, Boot Coffee and Café Imports and I’m sorry if I forgot anyone.

    Here's the group of rad roasters/guinea pigs:

  • Comments on this post (7 comments)

    • geoff watts says...

      This post made my heart smile. Bravo, Stephanie, for being so clear and honest and thoughtful in your writing. And for taking the time to share on a topic that is of great interest to everyone who loves coffee but that is all too often charged (npi) with a lot of hyperbole, stubbornness, dogma, and polarized opinion. You are right to ask so many open ended questions. We’ve all got so much to learn about roasting science, sensory evaluation, and how we can (or should) measure quality. Sometimes the definitions and metrics that emerge from our collective attempt to make sense of coffee quality end up feeling sort of obtuse or detached. It is nice to step back and think about why we score things a certain way, and how relevant those metrics are if the ultimate goal is maximum pleasure. Anyway, it is great to see that Roasters are finally getting their own competition that should serve to celebrate the artisanship, skill, and science that make roasting so freaking fascinating and so vitally important in the quality chain. Big props to everyone involved in making that happen, and good luck. And by the way I love the suggestion that the competition could evolve into a something where roasters are required to demonstrate different skills (using different kinds of coffees) and where success could be measured in a way that does not rely too heavily on a handful of judge’s individual definitions of beauty.

      September 28, 2012

    • Ryan says...

      Hey, could you repost a version of this on the RG forum so the rest of our peeps can check it out? Excellent post by the way. I think it’s really in line with most of our concerns as Roasters about the whole “competition” thing.

      September 28, 2012

    • Hanna Neuschwander says...


      Excellent food for thought in here. I recently wrote a book on roasters (a guidebook of sorts) on the west coast. For the book I interviewed over 60 roasters. It was amazing to me how ingrained the cultural preferences you describe are (and I think you’re right to call them “cultural”). Almost every smaller specialty roaster I interviewed, though not all of them, talked about “letting the coffee speak for itself.” Even though the roasting styles varied from hardcore Scandanavian style light roasts to full city, roasters used the same language to talk about how they weren’t intervening. (This in itself is problematic: We don’t have a good language for defining roast styles. “Light” and “dark” are too blunt. “Well developed” is too subjective. No one seems to like the old school “cinnamon” and “full city”, and besides that they seem too arbitrary and culturally specific—they wouldn’t translate well across national lines. But that’s a separate blog post.)

      This idea of “nakedness”, I think, has been a profoundly useful way of advancing the conversation about coffee and origin to this point, but I see its potential to undermine the good work of roasters going forward. It ignores reality: Roasting coffee fundamentally alters it. That’s the value the roaster adds. I don’t understand why so many folks are reluctant to own that fact. If all a roaster is doing is “not f*cking up” green coffee, why can’t I just buy some of the same high quality green from Sweet Marias and do the same thing myself? What good is a coffee roaster to me?

      In a number of cases, the roasters I talked with were quite young/new to the business. I think to a degree this line of argument had become a protective cover for the fact that they were still getting their roasting chops in order. It’s easier to lay your success or failure at the feet of your beans than it is the skill of your roasting. I drank a lot of coffees that were flat out poorly roasted. But I’ve watched every one of those roasters get better.

      I think roasting competitions could be a major way of advancing the overall project of roasting great coffee, of allowing roasters to differentiate themselves, and of encouraging innovation. And, as you suggest, of putting a finer point on some of the tricky questions we haven’t poked too many sticks at yet—the most essential of which is: how do you define the skills of roasting, and usefully measure them? Given the incredible curiosity at the heart of specialty coffee, defining some of these things will certainly drive younger and established roasters to become better, and will give them some useful tools for doing so.

      Hanna Neuschwander

      September 26, 2012

    • stephanie says...

      Thanks for your comments, guys. There is so much to talk about on this topic, hopefully we can keep it going!

      I definitely think there is a way to have great, bright acidity and balanced sweetness— but these things are also a matter of culture, taste and preference. What may be interpreted as “harsh” acidity to some, might be very balanced and wonderful to others. This competition really sparked my curiosity about various approaches to roasting across cultures and markets. Obviously, we have a lot of variation in what we do here in the states, which is interesting in itself.


      This Nordic Barista Cup video (you may want to skip ahead, the beginning is sort of boring) is interesting. The same coffees were roasted on six different machines, and the results were cupped. Obviously you can’t taste through the video, so you’re not really getting the whole experience, but the most interesting part to me was how very different the roasts were executed by different people. Very different placements of first crack, different finish times and temps and percentage loss—but all roasted to the same “level”. Some of this of course has to do with the machine you’re working with, and what makes sense with that equipment, but some of it is also style. How did everyone get to the approach they have to roasting? Was it taught? Based on experimentation? Based on equipment limits or advantages? There are so many approaches it seems that we’re a ways off from coming up with any “standard”, and even when and if that happens it will be broken and needs to be. On a related note, Jon’s blog on here “Coffee Under Pressure” addresses the same issue of standards in regards to espresso. Check it out.

      Keep it coming.

      September 20, 2012

    • Marshall Hance says...

      I roast light. Ten to fifteen degrees (Fahrenheit) post first crack. There’s such thing as sweetness in harmony with, even dominating acidity at this level that I’m not hearing echoed above. Either way, you great raise a question that’s on my mind: with so many ways to roast a great coffee, how is one approach judged better? I personally like the Nordic Roaster event that puts the attendees in charge of judging.

      September 19, 2012

  • Leave a comment

Success! Feel free to continue shopping or head to your cart .