Parameters for logging (fluentbit)
The logging feature is introduced in NetBackup 10.5 to to consolidate log files that have been distributed during the process of running NetBackup in a scale-out environment. To configure the fluentbit, ensure that the following prerequisites are met:
Ensure that the following system requirements and prerequisites are met before proceeding with the deployment:
System requirements: Cloud Scale deployment environment only
Application requirements: Cloud Scale fluentbit pods and containers:
Fluentbit collector pod and containers
Fluentbit DaemonSet sender pods and containers
Fluentbit sidecar sender containers
To view the current values, execute the command:
helm get values cloudscale -n <netbackup namespace> > new-values.yaml
Update the new-values.yaml file to modify the required parameters under the fluentbit section based on your environment and configuration requirements.
To apply the new values of fluentbit, execute the command:
helm upgrade --install cloudscale cloudscale-11.2.tgz -f new-values.yaml -n <netbackup namespace>
Table: Fluentbit collector configuration variables
Configuration value | Description |
|---|---|
fluentbit.volume.pvcStorage | This is the size of the PVC created for the fluentbit collector to store logs. |
collectorNodeSelector | This is how you can set the node selector for the fluentbit collector pod. |
Table: Log Cleanup configuration variables
Configuration value | Description |
|---|---|
fluentbit.volume.pvcStorage | This parameter describes the size of the PVC created for the fluentbit collector to store logs. |
fluentbit.cleanup.retentionDays | (number of days) - Number of days to retain logs before cleaning them up. For example1 would mean 1 day which would mean that if the logs were created the day before they are to be kept that 1 day and no cleanup would occur. The next day the logs are 2 days old and with a setting of 1 they would be cleaned up. Default: retentionDays: 7 |
fluentbit.cleanup.retentionCleanupTime | (hh:mm) - Time of day to cleanup the logs. This is based on the local Kubernetes time zone. The retention cleanup occurs once daily. Default: retentionCleanupTime: 04:00 |
fluentbit.cleanup.utilizationCleanupFrequency | (number of minutes) - Number of minutes to wait between subsequent utilization cleanups. This is based on start time of the previous cleanup, not when the cleanup finishes. For example: helm upgrade cloudscale cloudscale.tgz -n netbackup --reuse-values --set fluentbit.cleanup.utilizationCleanupFrequency=50 Default: utilizationCleanupFrequency: 60 |
fluentbit.cleanup.highWatermark | (number percent) - This is the percentage of storage utilization that the utilization cleanup will start cleaning up data one day at a time. It will continue until it reaches the low watermark or only the current days logs remain. If only current days logs remain it will start cleaning up individual files of the current day based on last updated time. For example: helm upgrade cloudscale cloudscale.tgz -n netbackup --reuse-values --set fluentbit.cleanup.highWatermark=80 Default: highWatermark: 90 |
fluentbit.cleanup.lowWatermark | (number percent) - This is the percentage of storage utilization that the utilization cleanup will stop cleaning up data once it gets below. Default: lowWatermark: 60 |
fluentbit.namespaces | This is a list of namespaces to collect stdout logs from. This is designed to allow you to filter out cluster logs and other logs not related to the NetBackup system in order to reduce network traffic and focus on relevant logs. Default: netbackup, netbackup-operator-system |
Table: Fluentbit DaemonSet configuration variables
Configuration value | Description |
|---|---|
tolerations | This is where you can set the tolerations of the daemonset pods to help determine which nodes they should be scheduled to. |
Table: Resizing Log Volumes
Configuration value | Description |
|---|---|
Resize Log Volumes: It is important that all the non-collector log volumes must remain the same size. This is because they are all using a common variable in the bp.conf file to determine when to be cleaned up and how much size to use. |
|
Table: Enabling the fluentbit sidecars logging flag
Configuration value | Description |
|---|---|
fluentbit.enableSidecarsLogging | Enables or disables sidecar log collectors used to gather NetBackup and VxUL logs in addition to standard Default value is false. Only Use the following command to enable sidecar logging: helm upgrade --install cloudscale cloudscale-11.2-4439002.tgz -n netbackup --reuse-values --set fluentbit.enableSidecarsLogging=true The flag may be toggled from false to true or from true to false after deployment/upgrade completes. Use the following command to get the current cloud scale values: helm get values cloudscale -n netbackup Use the following command to verify the kubectl get configmap nbfluentbitsidecarconf -n netbackup -o yaml |