Publishing

Why keep schema registries in sync

💡 Event schemas are important in many layers of your data stack. Keeping schema registries in sync across the stack can be a challenge, especially since stakeholders involved vary from non-technical folks with a preference for human readable tools like spreadsheets, to technical folks with a preference for computer readable schemas like JSON.

Publishing is a way to push your tracking plan specs from Avo into your other schema registries and downstream tools, including analytics platforms like Amplitude or Mixpanel, ingestion time validation tools like Segment Protocols, RudderStack or Snowplow, or connect to any tool or internal system via webhook. See publishing use cases and publishing integrations below.

Push or pull to keep schema registries in sync

Avo allows you to maintain a single source of truth for your schemas and keep your schema registries in sync, serving both your technical and non-technical stakeholders.

💡 Push or pull your schemas: To keep your Avo tracking plan in sync with your downstream schema registries you can either push your schema from Avo via Avo Publishing or pull your Avo schema via Avo CLI. We recommend leveraging push since you can configure that to make sure your downstream schema registries get updated when your main Avo schema gets updated.

1. Push: Publishing your schema: You can set up Publishing integrations where you publish your Avo schema into other schema registries. For example as JSON into a Webhook. We recommend leveraging auto-publishing for those integrations, where your schema will get published upon Avo branch merge (you can also manually publish a schema at any point). See use cases.

2. Pull: Pull JSON schema via CLI: An alternative to Avo Publishing integrations is pulling a JSON schema via the CLI. This way you can add a schema pull into a CI/CD. See docs to pull schema via CLI.

Publishing Integrations

Available integrations are:

• Segment Protocols
• Mixpanel Lexicon
• RudderStack
• Amplitude Govern
• Snowplow Data Structures
• mParticle Data Master
• Webhook

Publishing enables your Tracking Plan to be:

• Published whenever a branch is merged with Auto Publishing
• Published with a push of a button
• Downloaded from your Avo workspace on relevant format

Publishing use cases

See event and property descriptions from Avo inline in your analytics tool

Having the description at hand when navigating your events in your analytics platform, like Mixpanel or Amplitude, helps you more easily identify the events and properties you’re looking for when building charts and looking at dashboards. With an Avo publishing integration you can let Avo automatically push description and other tracking plan details into these tools, so you can have more context at hand when using the data.

Keep data validation in your CDP up to date with the tracking plan

Segment Protocols, mParticle Data Master, RudderStack Tracking Plans and Snowplow Iglu are all examples of CDPs with ingestion time validation. It allows you to validate the events as they come in, and for example route the bad events to alternative destinations. With Avo publishing you can make sure the schemas used for this validation are always up to date with your tracking plan in Avo, which increases the quality of your data in all downstream destinations. Being able to integrate with CDPs this way also allows your teams to gradually adopt Avo over time.

Receive a webhook every time the Avo tracking plan is updated

The publishing webhook is a building block that allows you to integrate your Avo tracking plan with your other schema registries and internal systems, tools and workflows. What can you build with the webhook? Here are some ideas to get you started:

• Update ingestion time validation:

  Automatically update your JSON Schema ingestion time validation whenever a tracking plan change is merged to the main branch. Making sure your pipeline is ready to approve the fresh events before they arrive.

• Manage tables in data warehouses:

  Automatically keep raw event table schemas in SnowflakeDB, BigQuery or Redshift in sync with the Tracking Plan – keeping your data streaming in smoothly

• Generate dbt models:

  Generate .yml config files describing your raw event tables in dbt. I'm also excited to explore the possibility of generating detailed dbt tests based on the Tracking Plan spec

• Automatically build dashboards in your visualization tool:

  The webhook contains detailed specs on the metrics you've defined in your Tracking Plan. Those specs could be used to automatically build out feature dashboards in visualization tools like Looker, Mode, Mixpanel or Amplitude

Setting up and configuring a publishing integration

Create a new publishing integration

To create a new publishing integration, find the Publishing part of the tracking plan in the left side navigation bar. From there you can click the "Add Integration" button to create a new publishing integration.

Then you can select the type of integration you want to create and give it a name.

Note that the publishing integrations are independent of the destinations defined in the "Sources" tab. However the publishing integration can use those destinations to filter which events should be included in the publishing integration. See more below in Filtering events for publishing.

Configure Auto publishing

When Auto Publishing is enabled all changes made on a branch will be published to the integration when the branch is merged.

Filtering events for publishing

You can customize which events in Avo are published to which integrations by using the events filter. Events can be filtered by Sources, Destinations and Tags. For example the following integration will only publish events sent from Web to Amplitude.

Filtering and source specific property presence

How is source specific property presence published?

• If you don't filter the integration by any source, the property presence is set to the "lowest common denominator" for all sources the property is sent from
• If you filter the integration with one source, the property presence is set to the presence configured for that source on the property
• If you filter by more than one source, the property presence is set to the "lowest common denominator" for the sources you're filtering by and are sending the property

What does the "lowest common denominator" mean?

• If the property is "Always sent" on all sources being published, the property presence is set to "Always sent" (required)
• If the property is "Sometimes sent" on one or more sources being published, the property presence is set to "Sometimes sent" (optional)
• If property is "Never sent" on all sources being published, the property is not included in the publishing

You can learn more on configuring when Properties are required or optional here.

Integrations

Segment Protocols

The integration to Segment Protocols allows for your Tracking Plan to be published into Segment Protocols whenever branches are merged in your workspace or with a push of a button.

Configure the Protocols integration

💡 The Protocols integration currently supports the established Config API. If you are only able to use the new Public API, please reach out to us.

To enable the direct integration with Segment Protocols, you need to configure it with the following parameters from Segment.

• Segment Workspace Slug: Found in the URL when logged into Segment: https://app.segment.com/{Segment workspace slug}/... or in the workspace settings
• Protocols Tracking Plan ID: Found in the URL of the tracking plan page: starts with rs_
• Segment Access Token: Create new API tokens for Avo and limit the scope of its permissions to ‘Protocol Admin’.

Publishing to Protocols

To publish your Avo tracking plan to Protocols, click the "Publish" button in the integration interface, or enable Auto Publishing to publish the events included in the publishing integration whenever a branch has been merged.

What Is Included In The Publish to Protocols?

Avo will override all events in the Protocols tracking plan that are included in the publish from Avo. Any changes made to an event in Protocols between Avo publish will be overridden on next Avo publish.

Avo will not override any events that are only defined in Protocols, or events that are not included in the Protocols integration in Avo.

Here's how the Avo Tracking Plan is mapped to Protocols:

Event

• Name: The event name as defined in Avo. If name mapping is applied to the event for a Segment destination, the name defined by the name mapping is used here.
• Description: This is the description as provided in Avo and list of triggers defined in Avo for each source
• Labels: All tags attached to the event in Avo will be published to Protocols as labels, on the key:value format "[Avo tag]:avo"

Event Property

• Name: The property name as defined in Avo. If name mapping is applied to the property, the name defined by the name mapping is used here.
• Description: The property description as defined in Avo
• Type: The property type and rules in Avo mapped to JSON Schema compatible rules
• Required: If the property is marked as "Always sent" in Avo it will be marked as required in Protocols

User Property

User Properties in Avo are published as identify traits to Segment Protocols.

By default, all user properties that are attached to events that are sent from at least one source in Avo, are included in the publish. When the publishing integration is filtered by source, only user properties sent with events from those sources will be included. To see exactly which user properties will be published in your integration, expand the "Payload Preview" section below the integration config.

Avo will not override any user properties (identify traits) that are only defined in Protocols, or user properties that are not included in the Protocols integration in Avo.

• Name: The property name as defined in Avo. If name mapping is applied to the property, the name defined by the name mapping is used here. Note that event specific name mappings will not be applied because user properties are not event specific in Protocols.
• Description: The property description as defined in Avo
• Type: The property type and rules in Avo mapped to JSON Schema compatible rules
• Required: If the property is marked as "Always sent" on all events in Avo, it will be marked as required in Protocols. Otherwise it will be marked as optional.

Group Property

Currently Avo does not publish properties into Protocols. If you'd like to publish group properties, please let us know, and we'll invite you to the beta when it goes live.

Category

Currently Avo does not publish categories into Protocols. If you'd like to publish categories, please let us know!

Mixpanel Lexicon

The integration to Mixpanel Lexicon allows for your Tracking Plan to be published into Mixpanel Lexicon whenever a branch is merged in your workspace or with a push of a button.

Configure the Lexicon integration

To enable the direct integration with Mixpanel Lexicon, you need to configure it with the following parameters from Mixpanel.

• Mixpanel Project Id: Found in the URL when viewing your project in Mixpanel: https://mixpanel.com/report/<Mixpanel Project Id>/...
• Mixpanel Service Account Username: See Mixpanel's documentation for creating a Service Account
• Mixpanel Service Account Secret: See Mixpanel's documentation for creating a Service Account
• EU Data Residency: Turn this on if you opted to store your data in the EU when you created your Mixpanel project See Mixpanel's documentation on data residency in EU

Publishing to Lexicon

To publish your Avo tracking plan to Lexicon, click the "Publish to Lexicon" button in the integration interface, or enable Auto Publishing to publish the events included in the publishing integration whenever a branch has been merged.

On publish, Avo will merge your Avo tracking plan with any existing events in Lexicon. Any changes made to an event in Lexicon between Avo publish will be overridden on next Avo publish.

Avo will not impact any events that are only defined in Lexicon, or events that have not been added to the Lexicon integration in Avo.

RudderStack Tracking Plans

The RudderStack integration enables you to publish your Avo Tracking Plan into RudderStack. You can configure the integration to publish whenever a branch is merged or with a push of a button.

Configure the RudderStack integration

To enable the direct integration with RudderStack Tracking Plans, you need to configure it with the following parameters from RudderStack.

• RudderStack Account Email Address: The email of a RudderStack account with Read-Write access
• RudderStack Tracking Plan Name: The display name of the RudderStack Tracking Plan you want to publish to
• RudderStack Access Token: Create new Personal Access Token for the account with the email address you provided above

Publishing to RudderStack

Avo will override all events in the RudderStack Tracking Plan that are being imported from Avo. Any changes made to an event in RudderStack Tracking Plan between Avo publish will be overridden on next Avo publish.

Avo will not override any events that are only defined in RudderStack, or events that have not been excluded from the RudderStack integration in Avo with filters.

Snowplow Data Structures

The Snowplow integration enables you to publish your Avo Tracking Plan into Snowplow. You can configure the integration to publish whenever a branch is merged or with a push of a button.

Configure the Snowplow integration

To enable the direct integration with Snowplow, you need to configure it with the following parameters.

• Snowplow Organization ID: Your organization ID issued by Snowplow, can be retrieved from the URL immediately following the .com when visiting the console, e.g. 13bbade6-1eff-42d0-861d-41cd3e52ac41 in https://console.snowplowanalytics.com/13bbade6-1eff-42d0-861d-41cd3e52ac41/contact-us.
• Snowplow Vendor Name: The vendor name of your Snowplow schemas, can be obtained in the Manage data structures section in the console.
• Snowplow Access Token: The secret access token to update your Snowplow data structures, can be obtained in the API keys section.

Publishing to Snowplow

Data structures interfaces are only compatible with pipelines that have been upgraded to use Iglu Server registries, rather than static S3 registries. Please check in your Snowplow console, to see if you need an upgrade or if your registries are ready to go.

The Snowplow API enforces a workflow of validating, testing on development and then deploying to production.

Avo publishing integration will take care of validating and deploying your data structures to the development environment, where you take over to test the data structures and promote the tested data structures to production.

Avo will override all events in the Snowplow Tracking Plan that are being imported from Avo. Any changes made to an event in Snowplow Tracking Plan between Avo publish will be overridden on next Avo publish.

Avo will not override any events that are only defined in Snowplow, or events that have not been excluded from the Snowplow integration in Avo with filters.

Note that only Admin users can promote data structures from Dev to Production. Learn more on how to promote the data structures with Snowplow Console here

Versioning

Avo will create new data structures with version 1-0-0 and will patch the existing data structures in your development environment on the same version as they are.

When promoting the tested data structures to production you will be required update the version. Read more about the versioning here

mParticle Data Master

The mParticle Data Master integration enables you to publish your Avo Tracking Plan into mParticle Data Master. You can configure the integration to publish whenever a branch is merged or with a push of a button.

Configure the mParticle integration

To enable the direct integration with mParticle Data Master, you need to configure it with the following parameters from mParticle.

• mParticle Client ID: Your client ID issued by mParticle. To get both Client ID and Client Secret press the person icon in the bottom left of your mParticle web UI. Then click "Settings" -> "API Credentials" and create new credentials. You'll get both Client ID and Client Secret there.
• mParticle Client Secret: Your client secret issued by mParticle.
• mParticle Workspace ID: The ID of the workspace containing your data plans. To find it click the app name in the top left of the mParticle web UI, then "Settings". You'll see the list of your workspaces, Workspace ID is the number below your workspace name.
• mParticle Plan ID: The ID of the Data Plan to update. To view available plans press "Data Master" -> "Plans" in the navigation panel in the mParticle UI. You'll see the list of your plans. Plan ID is the string below the plan name.

Publishing to mParticle

Avo will create a new version of your data plan in mParticle every time you publish. When publishing manually you can pick whether you'd like to overwrite existing mParticle data plan version or create a new one. Auto publish will always create a new version of the data plan in mParticle, by incrementing the version number of the most recent plan by one.

Avo will override all events in the mParticle Tracking Plan that are being imported from Avo. Any changes made to an event in an mParticle Tracking Plan between publishing from Avo will be overridden on the next Avo publish.

Avo will not override any events that are only defined in mParticle, or events that have not been excluded from the mParticle integration in Avo with filters.

Amplitude Govern

💡 Amplitude has recently closed it's public API making us unable to offer this option to our users. We will make it available as soon as they offer an API to update Govern.

The integration to Amplitude Govern allows for your Tracking Plan to be published into Amplitude Govern whenever a branch has been merged. You can also manually trigger a publish by clicking the "Publish to Govern" button on the integration screen or download a CSV file on the Amplitude Govern format, which you can manually import into Govern. The events to be published can be filtered by Sources, Destinations and Tags.

Configure the Govern integration

To enable the direct integration with Amplitude Govern, you need to configure it with the following parameters from Amplitude. They can be found from the project settings in Amplitude. From the left sidebar in your Amplitude workspace click: Settings > Projects > [Your Project Name]. The URL should look like this: https://analytics.amplitude.com/{org-name}/settings/projects/{project-id}/general.

• Amplitude API Key
• Amplitude Secret Key

Publishing to Govern

To publish your Avo tracking plan to Govern, click the "Publish to Govern" button in the integration interface, or enable Auto Publishing to publish the events included in the publishing integration whenever a branch has been merged.

What Is Included In The Publish to Govern?

On publish, Avo will merge your Avo tracking plan with any existing events in Govern. Any changes made to an event in Govern between Avo publish will be overridden on next Avo publish.

Avo will not impact any events or properties that are not defined in Avo, or events that have not been added to the Govern integration in Avo. Avo will not impact any custom events defined in Govern.

Here's how events, properties and categories are mapped from Avo to Govern:

Event

• Event type: The event name as defined in Avo. If name mapping is applied to the event for an Amplitude destination, the name defined by the name mapping is used here.
• Category: Amplitude Govern only supports one category per event, so the first category attached to the event in Avo is used here
• Description: This is the description as provided in Avo and list of triggers defined in Avo for each source

Event Property

• Event property: The property name as defined in Avo. If name mapping is applied to the property for an Amplitude destination, the name defined by the name mapping is used here.
• Event type: The name of the event this property is attached to. If the property is attached to multiple events in Avo, multiple properties will be created in Govern because Govern does not support reusing properties across multiple events with a global namespace.
• Type: The property type in Avo mapped to one of Govern's supported property types: string, number, boolean, enum or any
• Is array: true if the "list" checkbox is checked on the property in Avo
• Is required: true if the property is marked as "Always sent" in Avo
• Description: The property description as defined in Avo
• Enum values: If the property is of type string, and the "property matches..." rule is applied in Avo, the allowed values are provided here

Category

• Name: The name of the category as defined in Avo. If the category already exists in Govern, we reuse that one.

Govern Publishing Warnings

• 409 Conflict: Some events that are in your Govern Schema are not recognized by the Govern API, for example events that have been merged into other events in Govern or events that are blocked. Events that end up returning 409 error in Amplitude are likely not active. If you'd like to get rid of this warning please double check them and remove them from your Avo tracking plan if they are not active anymore.

Govern Publishing Errors

The following errors might occur when publishing your tracking to Govern. When publishing errors occur the error message in the Avo UI will contain which items could not be updated along with the error codes.

• 404 Not Found: This error can be thrown in few scenarios:

  • We tried to update an event that doesn't exist in your Amplitude Govern Schema
  • If this error only occurs for events and properties Amplitude hasn't seen yet, it might indicate that the schema in Amplitude Govern hasn't been initialized, or that you don't have access to the "Plan data" functionality in Amplitude Govern
  • If this error occurs for all of the events you are attempting to publish it might indicate that your Amplitude Govern Schema has not been initialized. Learn more about initializing your Amplitude Govern Schema in the Amplitude Govern docs. After you Govern Schema is initialized usually the API continues to return occasional 404 errors for a few more hours. If your Govern Schema has been initialized more than 6 hours ago and the error persists, please contact support.

• 408 Timeout: The publishing operation took too long. This can happen when publishing large tracking plans for the first time. Click publish again to resume the publish, it will continue from where it left off.

• 409 Conflict: We tried to update an item that doesn't exist in your Amplitude Govern Schema. We noticed that this error may happen if the entity has a non-default state in your Govern schema, i.e. it is marked as "Unexpected" or "Blocked". Try setting the state to "Expected" in Govern and unblock erroring entities and publish again. If that does not help this might be caused by an internal error. Please reach out to support so we can help you resolve the issue.

• 429 Too Many Requests: We've exceeded the number of Govern requests we can make per hour for the provided API credentials or there are too many concurrent requests to the Amplitude Govern API with given credentials. Please wait before trying to publish again and make sure that you are not running multiple publishing at the same time for the same credentials. You can learn more about Amplitude Govern request limits in the Amplitude Govern docs.

• Other error codes: Please contact us if you are experiencing other types of errors.

Webhook

The Webhook integration is a way to get your updated Tracking Plan distributed to an endpoint of your choice. This happens when a branch is merged or when the publish button is clicked in the integration interface.

Configure the Webhook integration

Webhook URL

To enable the Webhook integration you need to configure it with a URL where you want the updated tracking plan to be delivered.

The Tracking Plan will be delivered with a POST request containing the Tracking Plan on a JSON format. The only requirement for this endpoint is to return a 200 response status code when it has successfully received the tracking plan.

Payload Format

Currently there are three different payload formats you can choose from for the webhook: Snowplow Schemas, Avo Json, and Json Schema (default).

💡 To see a preview of the exact Webhook payload in your Avo workspace choose Publishing in the sidebar, then pick your Webhook from the menu and expand the Payload preview section

Snowplow Schemas

Json containing a Snowplow Schema for every event included in the Publishing integration. These schemas can for example be used to integrate Avo with Snowplow Iglu.

You can provide your own vendor value via the "Vendor" input in the publishing integration config.

Avo Json

Json representation of the state of your entire Avo workspace. Currently behind a feature flag. Contact us to enable.

Json Schema (default)

Json representation of your tracking plan, following the Json Schema standard. Example payload can be found here

Event Model

Rules Model

PropertiesRule Model

PropertyRule Model

NameMapping Model

Metric Model

Nested Event Model

groupByFilter

Where filter

Source Model

Destination Model

Category Model

PublishInfo Model