Reverse ETL

Reverse ETL allows the use of a database (aka: Segment Warehouse) as a source of data to be connected and sent to supported Segment Destinations. Previously, it was only possible to use a Segment Warehouse as a destination.

The structure of config varies with the value in strategy. All strategies will employ a key/value object with minimum depth of 1.

Strategy: "manual"

When strategy is "manual", config can be excluded altogether. With this strategy, syncs will never be triggered automatically, so they must be triggered manually through the UI or Public API route.

"reverseETLSchedule": {
  "strategy": "manual"
}

Strategy: "periodic"

When strategy is "periodic", the only supported config key is "interval", which will be a string that is in a format accepted by Go's time.ParseDuration function. For example, "3h" will sync every 3 hours, while "30m" will sync every 30 minutes. The interval must be greater than 10 minutes ("10m") and shorter than 1 week ("168h").

An example schedule would look like this:

"reverseETLSchedule": {
  {
    "config": {
      "interval": "1h"
    },
    "strategy": "periodic"
  }
}

Strategy: "specific_days"

When strategy is "specific_days", there are only 3 supported config keys: "days", "hours" and "timezone".

The "days" field must be an array of numbers, corresponding to the desired days of the week that syncs should run. Each item must be a number from 0 (Sunday) up to 6 (Saturday). There needs to be at least 1 item, but also no more than 7.

For example: [0, 6] would correspond to the weekend (Saturday and Sunday) while [1, 2, 3, 4, 5] would correspond to the weekdays (Monday through Friday).

The "hours" field must be an array of numbers, corresponding to the desired hours of the day that syncs should run. Each item must be a number from 0 (12am or 0:00) up to 23 (11pm or 23:00). There needs to be at least 1 item, but also no more than 24.

For example: [0, 12] would correspond to the running at midnight (12:00am / 0:00) and noon (12:00pm / 12:00).

The "timezone" field must be a valid value from the IANA database used by Golang, which should match the list found here. This allows the "hours" above to reflect your own time zone which may be easier to configure, but you can also choose "UTC" if you want to avoid things like DST (daylight savings).

A simple example config would look like:

"reverseETLSchedule": {
  "config": {
    "days": [1],
    "hours": [15],
    "timezone": "America/New_York"
  },
  "strategy": "specific_days"
}

A more complex example:

"reverseETLSchedule": {
  "config": {
    "days": [0, 6],
    "hours": [0, 6, 12, 18],
    "timezone": "America/Los_Angeles"
  },
  "strategy": "specific_days"
}

This would run syncs 4 times per day (midnight, 6am, noon, 6pm) but only on weekends (Saturday and Sunday).

Strategy: "cron"

Use the cron expression for precise scheduling. Remember to include only five fields, not six, as Segment's smallest time unit is minutes. If you need help, use a cron expression generator to create the ideal schedule.

Note: We do not allow cron expressions lower than 15 minutes.

Here is an example indicating 'At 15 minutes past the hour, every 3 hours, on day 1 of the month', e.g. (Tuesday October 01, 2024 at 12:15 am)

"reverseETLSchedule": {
  "config": {
    "spec": "15 */3 1 * *",
    "timezone": "America/Los_Angeles"
  },
  "strategy": "cron"
}