How To Handle Out Of Memory Exceptions During Export

Overview

In certain cases you may encounter an Out Of Memory exception during export. The reason is usually the fact that the survey contains a too large number of variables.

The most common cause to this is the fact that nested loops have been configured during scripting by the scripter. Nested loops may generate a very large data set as the variables number is a multiplication of the internal question variables.

Each variable of each question inside the loop chapter will start with the iteration index - "I_NumOfIteration" and then the variable name as it is defined to be exported.

Example: If the loop is iterating a maximum of two iteration and inside the loop chapter I have a single choice question with a variable name: Q_1 then the exported iteration data for this question will have these two names: I_1_Q_1 and I_2_Q_1

For Nested Loops you will have an additional nesting of variables for example   I_1_I_1_Q_1, I_1_I_2_Q_1.... and I_2_I_1_Q_1, I_2_I_2_Q_1...

The outcome of trying to export such a large data set would normally result in an Out Of Memory exception as can be seen in the following screenshot:

 

If the survey has not yet been moved to Production (so no production interviews have been collected with it yet) we recommend a review of the nested loop structure and a redesign to ensure a reasonable data structure size. Contact our support team if you need assistance with understanding how to adjust the script. 

Important Note: It is a recommended best practice to generate test interviews (manually or using the dummy Data Generator) and review the generated data structure before starting field work as your data processing team may find out that the data structure doesn't meat their expectations (structure or size) so the script will need modifications. 

 

If data has already been collected there are several possible ways to reduce the exported variables: 

1. The most common way is to check the 'do not create empty variables' in the Options tab of the export process:
    
   
   This will not include the exported data variables that have a missing value for the entire data structure.
   
   
2. Change the missing value to be an empty string rather than a value (for example -1) as this may reduce the data structure size is a large part of the data structure is composed of null responses.  

Note: There are several other possible techniques and please open a support request with the support team if the above has not resulted in you being able to successfully export the data.