Export Data from Netcore to Databricks
Send Netcore CE events to your Databricks workspace.
Overview
A Databricks export sends Netcore CE event data to a Databricks table. You choose the event, the fields, any changes to apply, and a schedule. The export has four steps: Setup, Data selection, Data customisation, and Schedule.
Pre-requisites
- A Databricks account with a service principal. You need its Client ID and Secret key.
- The Server hostname and HTTP path of your Databricks SQL warehouse.
Connect Databricks for Export
You create the connection once. Then you use it in your exports.
- Log in to the Netcore CE dashboard and navigate to Settings > Integrations.
- Under the Data warehouse category, select Databricks. An Integration panel opens.
- Under Integration type, select Export to Databricks.
- Click ADD INTEGRATION.
- Fill in the Basic details. See the table below.
Export Data to Databricks
- Click ACTIVATE. The button stays disabled until you complete the required fields.
| Field name | Description |
|---|---|
| Connection name | A unique name for this connection. Up to 100 characters. |
| Authentication | OAuth M2M. Click View setup instructions for help. |
| Client ID | The client ID of your Databricks service principal. |
| Secret key | The OAuth secret of your Databricks service principal. |
| Server hostname | The hostname of your Databricks SQL warehouse. |
| HTTP path | The HTTP path of your Databricks SQL warehouse. |
A Steps to follow list on the same page repeats the setup notes. For where to find each value in Databricks, see Databricks.
Start a Databricks Export
- Log in to the Netcore CE dashboard and navigate to Settings > Data transfer.
- Click the Export tab, then click CREATE NEW. A panel titled "How would you like to start?" opens.
- Select Export data.
- Under "What would you like to export?", select Events.
- Click NEXT. A panel titled "Select a destination to export your data" opens.
- Click Databricks.
Start a Databricks Export
The Databricks event export wizard opens. Click FINISH LATER at any point to save your progress.
Step 1: Setup
| Field name | Description |
|---|---|
| Export title | A unique name for this export. Up to 80 characters. |
| Connection | Select your Databricks export connection. If none exists, click the Connection link to set one up. |
| Path | Select the path to use. This field activates after you select a connection. |
| Catalog | Select the catalog that receives the events. |
| Schema | Select the schema in that catalog. |
| Table | Select the table that receives the events. |
| Alert email | Select the email addresses that receive alerts about this export. |
After you select a table, click PREVIEW to see sample records. Use it to confirm that you picked the right table.
Click NEXT. Click Learn more about Databricks data export at the top right to open the help page.
Step 2: Data Selection
This step controls what data Netcore CE sends. The Data preview panel on the right lists the fields. It updates as you make changes.
- Under Events, select Engagement or Behavioural. Then select the events.
- Select the Event schema: Standard or Consolidated.
- Review the Standard field(s). Netcore CE adds them by default.
- Optional: Turn on Additional details and select the fields to include.
- Optional: Turn on Filters and select the values to filter by.
Click NEXT.
Step 3: Data Customisation
This step is optional. Use it to change the data before Netcore CE sends it.
- Turn on the Transformation toggle.
- Click + ADD to create a rule.
- Select the transformation and the target field.
| Transformation | What it does |
|---|---|
| Mask field | Hides the value of a sensitive field. |
| Rename column | Changes the column name. |
| Field to string | Converts the field to a string. |
| Date format | Changes the format of a timestamp. |
Click SEND TEST DATA to send a sample and check your setup. Then click NEXT.
Step 4: Schedule
- Select the start date and time.
- Select the end date and time. The export stops after this time.
- Select the Frequency, for example Daily, Weekly, or Hourly.
- Click SAVE or FINISH to activate the export.
Export Status and Monitoring
The Export tab of Settings > Data transfer lists every export. Each row shows the export title, its ID, a status badge, the destination, the frequency, and the total transfers.
Click the three dot menu on a row to open actions such as Duplicate, Pause, and Stop.
FAQs
How do I export Netcore CE events to Databricks?
Connect Databricks with the Export to Databricks integration type. Then go to Settings > Data transfer > Export, click CREATE NEW, and select Export data, Events, and Databricks. Complete the four steps.
Which authentication method does the export use?
The export uses OAuth M2M. You provide the Client ID and Secret key of a Databricks service principal.
Which event types can I export?
You can export Engagement and Behavioural events.
Can I change the data before it reaches Databricks?
Yes. In the Data customisation step, mask a field, rename a column, convert a field to a string, or change a date format.
Can I test the export before I start it?
Yes. Click SEND TEST DATA in the Data customisation step.
Can I schedule the export?
Yes. Set a start time, an end time, and a frequency in the Schedule step.
Can I save my progress and finish later?
Yes. Click FINISH LATER at any point in the wizard.
Related Pages
Updated 1 day ago