Skip to content

Running Jobs

Overview

A user might face performance issues when running a job on the platform. Performance issue is defined as any job taking more time than expected.

The user should follow the below best practices to avoid facing job performance issues.

User inputs

Sample size:

  • To run experimental jobs, select a small Sample Size (for example: 1000). This helps in understanding the issues in the object definition and fix them quickly.

  • To run jobs for approval, select a large Sample Size (as recommended by the Modelling Head). This job is expected to take more time.

  • If there is already a job for any object on the platform, which is using a super-set of input tables wrt the input tables used in the current object (for example, if the user wants to run a simulation for a Data Element and if there is already a model job run where this DE is an input, the model would be using the super-set of tables required by the DE), the user can select the job of the object having super-set of tables, which will have a significant reduction on the current object's job runtime (if table filtering is a major component for a job).

Subsetting Criteria:

  • Ensure each input table is filtered as much as possible. The user can add table filtering conditions so that only the required input data is read from the data lake. If there is a column on which a given table can be filtered.

  • User can able to write more advanced Expression in the Sub-setting Criteria when they run simulations. Using expressions in sub-setting criteria means applying conditions or filters that include arithmetic operations or function calls.

Example: 

date_to_str(transaction_date) > "2020-01-01"


month = per_num % 12
year = int(per_num / 12)
monthyear = year * 10000 + month

save

  • If table filters are not provided in Subsetting criteria, it will result in a large amount of data being read from the data lake, leading to large job runtime.

Large number of jobs:

  • A large number of jobs (across the platform) might cause a lot of queuing time for a given job. The queuing time will be shown separately in the Job Details page (in runtime popup) and the job will move to RUNNING state, only when it actually starts executing.

  • The Job Board can help in understanding the user’s job position in the execution queue.

save

Platform Team Support

After following the above best practices, if the job is still not expected to take as much time as it is actually taking, then the user should raise the issue with the platform support team for Platform/Infra level issue with all the required information:

  • Object information: Name, Id, Type (DataElement/Feature/Model)

  • Job information: Name, Id, StartTime, Sample size, Job Configs (if any), Subsetting criteria for each table, Logs (if any), link to the job.