Configuring advanced dynamic multi-streaming parameters
You can configure these optional parameters in the backup host.
Table: Advanced parameters for backup host
Parameter | Default/ Configuration | Description |
|---|---|---|
COSP_LIST_API_ MAX_OBJECTS | Default: 5000 for Azure backups and 1000 for AWS/S3 backups. Configuration:
| During object listing, this value is used as the batch size in the ListObject API. Reducing this value may slow down the overall backup process. It is recommended to set it according to your cloud provider's specifications or recommendations. |
COSP_DOWNLOAD_ OBJECT_PART_ SIZE_IN_MB | Default: 16 Configuration: 1 - 512 | Represents the size of each chunk or part used when downloading large objects in multiple chunks. If the cloud is fast and capable of handling larger chunks efficiently, increasing this value may improve the overall backup speed. |
COSP_TOTAL_ OBJECT_DOWNLOAD_ WORKERS | Default: 160 Configuration: 10 - 1000 | This parameter determines the number of parallel tasks used to download objects from a specified bucket. If the cloud is slow, increasing this value can cause issues such as request failures or retries. However, if the cloud is fast, increasing this value can speed up the backup process. It's also important to note that increasing the number of workers consumes more CPU resources. Since this value applies to each backup job, it's crucial to configure it carefully, especially when running multiple parallel jobs. |
COSP_STREAM_ CHANNEL_BUFFER_ SIZE | Default: 100 Configuration: 100 - 5000 | This parameter restricts the total buffer size/number of objects to keep in the cache for processing after listing. If multiple streams are running, then this is the buffer/cache size of each stream. |
COSP_DB_INSERT _BATCH_SIZE | Default: 500 Configuration: 500 - 5000 | Represents the number of downloaded objects that are sent to bpbkar for final processing. Increasing this value to an optimistic number, such as 1000, can boost the overall backup speed, provided the staging location has sufficient space to handle the batch size. However, if the staging location lacks enough space, increasing this value may slow down the backup process. |
COSP_STAGING_ LOC_WATER_MARK_ IN_PERCENTAGE | Default: 80 Configuration: 30 - 80 | This parameter restricts the usages of the staging location. It accepts the value in percentage. The default value is 80. The range is 30 to 80. For example, COSP_STAGING_LOC_ WATER_MARK_IN _PERCENTAGE= 70 |
COSP_SPACE_ MANAGEMENT_TIMEOUT _IN_MIN | Default: 5 Configuration: 5 - 60 | The time to wait before starting a backup, if the required space is not available on the temporary staging location. The value is in minutes. Default value = 10. The range is 5 to 60 (minutes). For example: COSP_SPACE_ MANAGEMENT_TIMEOUT _IN_MIN = 20 |
You can configure this optional parameter in the primary server.
Table:
Parameter | Default/ Configuration | Description |
|---|---|---|
DYNAMIC_STREAMING _START_CHILD_BACKUP _JOBS_TIMEOUT | Default: 600 (Seconds) Configuration: 600 - 3600 (Seconds) | This timeout value determines how long each parent job waits to start all its child jobs, when multiple jobs are triggered with multiple streams. If the timeout is reached, the job fails with a crawler timeout error. We recommend setting this value as high as possible if multiple backups are running. Note that this setting takes effect only before running a backup job. |