Jobs
A job defines how the Extract and Load (EL) process is performed between your data sources and your data warehouse. Each job is always linked to a specific table or endpoint, depending on the type of connector used.
Overview
A job includes various configurations that define the EL process, such as:
Sync type: Specifies how the data will be extracted from the source and written to the target.
Sync schedule: Determines how often the job will run (e.g., hourly, daily).
Column Selection: Allows you to choose which columns will be included in the data extraction.
Within a connection, each job can be configured independently, allowing for significant customization and adaptation to your specific use case.
This flexibility ensures that data workflows can be precisely tailored to meet different requirements. However, in API-based connectors, certain jobs may have dependencies on others due to the nature of the API's structure or limitations.
Dependent Jobs
Dependent jobs occur when information from one job is required for the execution of another.
Example Scenario: Consider a CRM system with two endpoints:
/deals: Lists all deals, providing IDs for each./deal-details: Requires the deal IDs from the first endpoint to fetch detailed records for each deal.
In this case, the Deals job must execute first to gather the necessary IDs, which are then passed to the Deal Details job for its execution.
Key Characteristics of Dependent Jobs:
Execution Order: Dependent jobs always run after their parent job has completed successfully.
Sync Type Flexibility: While the jobs are dependent, the parent and child can have different synchronization types. For example, the parent job might use incremental sync, while the child job performs a full refresh.
This approach ensures accurate data flow between jobs while maintaining flexibility in how each job operates within its dependency hierarchy.
How to create a job
Specify the Columns to Synchronize
To select the columns to be synchronized within a Job, follow these steps:
Navigate to "Manage Jobs": Go to the job management section in the platform's interface.
Open the Column Selection Modal:
Locate the desired job in the list.
Click the "Open" button associated with that job to access the column configuration modal.
Customize Column Selection:
Select All: Choose to include all available columns from the data source.
Choose Specific Columns: Select only the columns relevant to your use case.
(Optional) Anonymize Columns: Configure columns to be anonymized if sensitive data needs to be protected during synchronization.
Last updated