Anyone know how to calculate reliability when each item that has been rated could have more than one possible code? E.g., not just rated as "yellow" or "red" but could be "yellow" and "blue", if that makes sense. I usually use Cohen's Kappa but this does not work for items that can have have more than one possible code.
Cheers!