Coding comparison query

The coding comparison query compares coding by two users to measure 'inter-rater reliability'—the degree of coding agreement between them. Agreement is measured by two statistical methods:

• Percentage agreement: the number of content units on which coders agree (to code or not to code), divided by the total number of units—reported as a percentage.
• Cohen's kappa coefficient: a statistical measure that takes into account the amount of agreement expected by chance—expressed as a decimal in the range –1 to 1 (where values ≤ 0 indicate no agreement, and 1 indicates perfect agreement).

You can select specific codes (or relationships or sentiments) or cases for comparison, in selected data files, datasets, externals and/or memos. You can also select groups of codes or files by identifying folders, static sets or classifications they belong to.

It is possible to compare coding between groups of users, however, standardly, comparisons are made between individual coders. If comparing groups, content is taken to be coded if at least one member of a group coded it. Coders are identified by their NVivo user profiles User profiles.

Both text coding and region coding can be compared. These are treated separately, producing separate results:

• All coding in documents, datasets, externals, memos, codes and cases is text coding.
• All coding in picture, audio and video files is region coding, using pixel ranges (rectangles defined by their top-left and bottom-right corners) or timespans. Note: 
  • If you code transcript text in an audio or video file, the timespan of the entire transcript row is treated as having been coded. So, for example, two coders who code different sentences in a row will be reported as in agreement for the entire row timespan.
  • If you code text in the description/notes of a picture file all the pixels in the picture are treated as having been coded.
• PDFs can have both region (pixel range) and text coding.

Text coding uses text characters as the unit of comparison.

You can save query configuration settings to run the same query at a later time, when more coding has been done. Queries are saved under Queries / Query Criteria in the Navigation View.

You cannot save query results in NVivo, however you can export them to other applications (e.g. Excel) to save. Export query results

Run a coding comparison

1. On the Explore tab, in the Queries menu select Coding Comparison. The Coding Comparison Query dialog box opens.
2. For User Group A and User Group B, select the users whose coding you want to compare (recommended to compare individual coders). If you include more than 1 user in a group, any coding carried out by any of the users in the group counts towards the comparison.
3. Use the At field to select the codes and/or cases you want to compare coding for:
   • All Codes: coding to all codes and cases in the selected files
   • Selected Codes: coding to selected codes and/or cases in the selected files
   • Codes and Cases in Selected Static Sets: all the codes and cases included in selected static sets
   • Cases Assigned to Selected Classifications: all the cases with selected case classifications
   • Codes and Cases in Selected Dynamic Sets: all the codes and cases included in selected dynamic sets

   NOTE: If you include an aggregate code in the scope of a query, content coded to it or its direct children is included in the results. Aggregate codes

4. In the Scope field select the files in which you want to compare the coding:
   • Files & Externals: all data files and externals, but not memos
   • Selected Items: selected data files, externals and/or memos
   • Items in Selected Folders: all data files, externals and/or memos in selected folders
   • Files, Externals & Memos in Selected Static Sets: all data files, externals and memos included in selected static sets
   • Files Assigned to Selected Classifications: all data files, externals and memos with selected file classifications
   • Files, Externals & Memos in Selected Dynamic Sets: all the data files, externals and memos included in selected dynamic sets
5. Select Display kappa coefficient and/or percentage agreement to include in the results (you must select at least one).
6. Select Text coding and/or Region coding for the type/s of coding you want to compare:
   • All coding in documents, externals, memos, codes and cases is text
   • All coding in picture, audio and video files is region coding, using pixel ranges or timespans
     
   • PDFs can have both region (pixel range) and text coding.
7. To save the query settings, check Save Criteria at the top of the dialog box. Name the query and optionally provide a description.
8. Click Run.

Query results are displayed in the Detail View (see below).

Coding comparison results

Results for text and region coding are shown on different tabs in the Detail View.

Each row shows data for one code or case in one file. Results for single codes or cases across all files in the query are not shown, nor an overall value for all codes or cases and files. You can calculate these independently by exporting the results data Calculating across multiple files and/or codes).

To view the content that a row in the results table refers to, right-click in the row and select Open Code/Sentiment/Relationship/Case or Open File .

1 The code, sentiment, relationship or case being compared.

2 The name of the file, and its location, where the code or case was coded.

3 The file size, measured as follows:

• Documents, datasets, memos and externals  = number of characters
• PDFs = number of pages and number of characters
• Media file = duration in minutes/seconds/tenths of seconds
• Picture = the total number of pixels expressed as height multiplied by width

4 The kappa coefficient—shown only if you selected Display Kappa Coefficient.

5 The green columns show percentage agreement (shown only if you selected Display percentage agreement):

• Agreement = sum of columns A and B and Not A and Not B
• A and B = the percentage of content coded to the selected code by both Group A and Group B
• Not A and Not B = the percentage of content coded by neither Group A nor Group B

6 The red columns show percentage disagreement (shown only if you selected Display percentage agreement):

• Disagreement Column = sums of columns A and Not B and B and Not A
• A and Not B = the percentage of content coded by Group A and not by Group B
• B and Not A = the percentage of content coded by Group B and not by Group A

How is percentage agreement calculated?

NVivo calculates percentage agreement for each combination of code or case and file.

Percentage agreement is the percentage of file content (measured in characters, pixels or tenths of seconds), on which the two users agree that it should be coded to a specific code or case, or not.

For example, in a document with 1000 characters where:

• 800 characters have not been coded by either user
• 50 characters have been coded by both users, and
• 150 characters have been coded by only one user

the percentage agreement is (800 + 50) ÷ 1000 = 85%, because both users 'agree' about 850 of the characters. (For pictures or PDF regions, pixel ranges are used instead of characters, and for media files, tenths of seconds.)

How is the Cohen kappa coefficient calculated?

Cohen’s kappa is widely used to quantify the level of inter-rater agreement between two raters (i.e. coders). The formula calculates the agreement between two coders and then adjusts for agreement that would happen by chance.

The formula is: κ = P₀ – P_e / 1 – P_e
where P₀ is the amount of agreement between two coders (equivalent to the 'percentage agreement' calculated by NVivo) and P_e is the probability of chance agreement.

The formula can be illustrated by the following table, where:

• P_yy is the proportion of content that both coders assigned to a code
• P_yn is the proportion that coder 1 assigned to the code and coder 2 did not
• P_ny is the proportion that coder 2 assigned to the code and coder 1 did not
• P_nn is the proportion that neither coder coded to the code.

The sum of these proportions is 1: P_yy + P_yn + P_ny + P_nn = 1

The observed agreement, P₀ is: P_yy + P_nn

The probability of chance agreement, P_e is: (P_yy + P_yn) × (P_yy + P_ny) + (P_ny + P_nn) × (P_yn + P_nn)

Interpreting kappa coefficients

If two users are in complete agreement about which content to code in a file, then the kappa coefficient is 1. If there is no agreement other than what could be expected by chance, the kappa coefficient is ≤ 0. Values between 0 and 1 indicate partial agreement.

Different authors have suggested different guidelines for interpreting kappa values, for example (from Xie, 2013):

Kappa vs. percent agreement

Kappa values can be low when percentage agreement is high. For example, if two users code different small sections of a file leaving most content uncoded, the percentage agreement is high, because there is high agreement on content that should not be coded. But this situation is likely to occur by chance (i.e. if the coding was random), and so the kappa coefficient is low.

Conversely, if most of a file is not coded but there is agreement on the content that is coded, then percentage agreement is again high, but now the kappa value, too, is high, because this situation is unlikely to occur by chance.

All kappa coefficients are 0 or 1

If all the kappa values in a query are 0 or 1 it may indicate that one of the two users being compared has not coded any of the selected files to the selected codes, i.e. you may have selected the wrong files, codes, or coders for the query.

If one user’s work has been imported from another project it may indicate that their coding was not imported. When merging projects with the intention of running coding comparisons, ensure that all documents and codes in the projects (including coding structures) match properly. When configuring import:

• select to import 'All' project items, or 'Selected (including content)' with 'Coding' selected.
• select 'Merge into existing item” for duplicate items

See Merge/import projects

Calculating across multiple files and/or codes

NVivo calculates percentage agreement and kappa coefficients for each combination of code or case and file. It does not calculate values for a single code or case across all the files in a query's scope, nor overall values for all the codes/cases in all the files. To get these values, export the results into another application, such as Excel, to calculate yourself Export query results.

Before calculating the additional values decide how you want to weight the files—that is, treat each file equally or weight them according to the amount of codable content they contain.

To help understand the calculations, download the Coding Comparison Calculation Examples spreadsheet, which has four worked examples using spreadsheet formulas:

• Average figures for a single code across 3 files (weighting each file equally)
• Average figures for a single code across 3 files (weighting each file according to its size)
• Average figures for 5 codes across 3 files (weighting each file equally)
• Average figures for 5 codes across 3 files (weighting each file according to its size)