All data that flows through our system will be delivered right where you want it. The specific data delivery option you choose is defined in the rules you’ve created. We offer several data destination options that fall under the categories of “Queues” and “Destinations”.
Queues are the default data delivery choice for most customers. Powered by RabbitMQ, they allow your application to pull data directly from the queue, which is useful for near-real-time event notifications and data flows. In the event of downtime for your application, our data queues hold events for 48 hours before discarding them, giving you time to get back online without data loss. Refer to this tutorial for more information on creating Queues.
Our platform can push data directly to cloud storage solutions like Amazon DynamoDB and Google BigTable, and to Keen or Webhooks. For these Destinations we use buffers to guarantee data delivery. Data that is not accepted by the Destination is marked as delayed and we’ll attempt delivery again on a back-off schedule. If data is still not accepted after 120 hours, the destination is cancelled, and buffered data is dropped.
Important Considerations for Webhooks
Webhooks are intended to be used for low-volume data delivery that is not mission critical. As of our December 10, 2019 release, we began guaranteeing data delivery using a 120 hour buffer. Our original implementation of Webhooks, where URLs were written directly into Rules, did not have guaranteed delivery has been deprecated. Support for this method will end on June 1, 2020. Please contact us if you require assistance moving to the new Webhooks Destination.
Tips for Working with Queues
Although you can create Queues from your RabbitMQ client, we strongly recommend you use the Dashboard application to create queues to ensure correct bindings and to manage your queues via our interface.
Visit RabbitMQ for a list of client and developer tools.
When you've selected a RabbitMQ client, see our Queue Connections page for how to consume from your queues.
Monitor your Queue consumers to make sure they are active so you don't lose any data to the 48 hour buffer limitation. Follow this guide to set up a monitor with AWS Lambda.