@jabiko thanks for the note. Glad our product is working so well for you.
re:Azure we are working on some new things :) . Feel free to drop me a message if you'd like to discuss further (ramon@tigerdata.com).
100% agree. OpenTelemetry solves half of the problem by standardizing and unifying instrumentation. With Promscale our aim is to solve the other half with a unified storage and query language for metrics, traces and logs. This release is a big step towards realizing that vision. It includes:
- Integrated with Grafana for dashboarding as well as for the tracing experience. Note that Grafana offers a very similar experience to Jaeger for traces (they reused the Jaeger code) and we extend that experience with a set of dashboards that deliver additional visibility into your services (the blog post has more details about those)
- SQL for querying and correlating traces and metrics.
So pretty much everything you said except logging which is in our roadmap.
That's definitely our goal: to offer a solution that is very easy to self-host that covers most of the features APM products provide within Grafana so you don't have to go back and forth between different tools. And a key advantage is that you can customize the experience and extend it in any way you want because it's made of editable Grafana dashboards populated with data from SQL queries.
We are also super excited to offer this capability to all our TimescaleDB users.
Feel free to ping us on our community Slack (https://slack.timescale.com #promscale channel or drop me an email ramon[at]timescale.com). We would be happy to answer any question you have, learn more about your needs and get your feedback about the product.
The link above is the right slack group, but you need to go in through the URL I provided to get the right invite tokens. Sorry, Slack isn't fully designed as a community platform, even though many use it for such.
SQL definitely opens lots of possibilities to analyze the data. And lots of visualization tools already integrate with PostgreSQL/TimescaleDB which Promscale uses to store data so you have a lot of options to pick from.
Would you mind sharing some of the problems you've had to use SQL for?
1. Find a trace where there is a span of type A with tag `xyz: true`, that has a direct child span of type B with tag `abc: 42`, that has at least one non-direct child span of type C with tag `lmn: forty-two`. So basically complex queries where I'm looking for a trace while making assertions about tags in different spans, possibly spans that are very far from each other in the trace (non-direct child, just joining on trace_id in that case, not child.parent_id = parent.span_id).
2. Analytical performance analysis across all spans of a certain type in a specified time span (a week for example). I have spans of type A, each of these spans has multiple children of type B, now I want to make some calculations involving the duration of A, the count of B, durations of B, and then slice and dice (group by) this by specific tags in those spans.
Thanks for sharing. Makes total sense. As soon as you have to query across span attributes AND span parent/child relationships you need a more sophisticated query language.
As @akulkarni said, Promscale supports Prometheus metrics and OpenTelemetry traces natively and there are different ways to correlate both signals. I am actually delivering a talk that goes over the different ways you can correlate Prometheus and OpenTelemetry data tomorrow at Prometheus Day :)
One is via adding exemplars to your Prometheus metrics that link to specific traces that are representative of the value of those metrics. In Promscale you can store all that information and then display it in Grafana as explained in 1. That's the way that is most often discussed but typically involves deploying one backend for metrics and one backend for traces instead of just one as is the case with Promscale.
With Promscale you can also correlate metrics and traces using SQL joins. That opens a whole set of new possibilities. For example, imagine if you could retrieve the slowest requests happening on services running on nodes where CPU usage is high to understand how they are impacting their performance. Or imagine you are seeing a specific OOM error often in your traces and you could run a SQL query to look at the evolution of memory usage in the last 24 hours of nodes where those OOM errors are more frequent to see if you spot anything strange happening. You could even go a step forward andretrieve in the same query what processes are consuming the most memory in those nodes to pinpoint the processes that could be causing the issue.
I'm not a server side person but from my experience, other types of data app developer might want to join are various kinds of product-specific data, like feature flags or user-specific dimensions for each request. These aren't typically in the trace data itself.
These are different from what product-agnostic "performance engineers" tend to look into, so I understand if this is out of scope. Although I think product people should look into these numbers as well, instead of just throwing them into the performance team's plate :-/
I should have mentioned that correlating observability data (or sometimes product metrics collected via Prometheus) with product data (or any other data really like business data) can be super useful and totally possible with Promscale because PostgreSQL is under the hood. So you could have a copy that data into the same PostgreSQL instance used by Promscale or maybe use Foreign Data Wrappers (1). This would allow you to analyze, for example, api request latency by product plan the customer is subscribed to or based on which feature flags are enabled for their account, etc. without having to add all those attributes as labels to all your metrics which can be technically complex and also costly.
We actually do this within Timescale Cloud, and it's amazing.
It allows us to cohort performance data across data stored in others microservice databases (e.g., by account types, projects, billing data, etc.). JOINs across foreign data wrappers using TimescaleDB + Postgres, all within the database and without ETL or application code needed.
So you could look at Prometheus data for your trialers vs. customers, for customers running more than X services, for customers that pay more than $X per month or have been a customer for more than 6 months, etc.
It's super useful across operations, support, product, customer care, and more...
Hey rudasn, I'm Ramon, the author of the blog post.
You can definitely use the tools mentioned in the blog post on bare metal as well. All components in the demo environments described in the blog post run on Docker on a host, no Kubernetes or cloud services. It would also be possible to run them directly on the host with no Docker.
Usually the challenge is that many of those tools have have been designed with microservices and cloud native environments in mind. So typically they are easier to set up in those environments and most of the documentation you'll find online is about those types of setups.
