{"id":30309,"date":"2022-06-17T02:35:33","date_gmt":"2022-06-17T02:35:33","guid":{"rendered":"https:\/\/www.devopsschool.com\/blog\/?p=30309"},"modified":"2022-12-23T05:52:43","modified_gmt":"2022-12-23T05:52:43","slug":"datadog-setting-up-ecommerce-datadog-observability-katacoda-simplified-steps","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/blog\/datadog-setting-up-ecommerce-datadog-observability-katacoda-simplified-steps\/","title":{"rendered":"Datadog – Setting up Ecommerce Datadog Observability katacoda simplified steps"},"content":{"rendered":"\n

<\/h4>\n\n\n\n

Step 1 – Spinning Up Our Legacy E-Commerce Shop<\/h1>\n\n\n\n

Our legacy monolith shop uses Ruby on Rails and Spree. We’ve started to build out a first set of microservices, and these have been added to an initial set of containres.<\/p>\n\n\n\n

We use docker-compose<\/code> to bring it up and running. There’s a prebuilt Rails Docker container image, along with the new Python \/ Flask microservice which handle our Coupon codes and Ads which display in the store.<\/p>\n\n\n\n

In this workshop, we’re going to spin up and instrument our application to see where things are broken, and next, find a few bottlenecks.<\/p>\n\n\n\n

We’ll focus on thinking through what observability might make sense in a real application, and see how setting up observability works in practice.<\/p>\n\n\n\n

Our application should be cloned from Github in this scenario, and if we change into the directory, we should be able to start the code with the following:<\/p>\n\n\n

$ cd \/ecommerce-observability\n$ POSTGRES_USER=postgres POSTGRES_PASSWORD=postgres  docker-compose up<\/code><\/span><\/pre>\n\n\n
Once our images are pulled, we should be able to jump into and view the application within Katacoda:<\/p>\n\n\n\n
https:\/\/2e652dae321844ddb7f22fd05609a510-167772165-3000-ollie02.environments.katacoda.com\/<\/a><\/p>\n\n\n\n
Try browsing around, and notice the homepage takes an especially long time to load.<\/p>\n\n\n\n
\"storedog\"\/<\/figure>\n\n\n\n
The first thing we’ll do is see where that slow load time may be coming from.<\/p>\n\n\n\n
<\/h4>\n\n\n\n
Step 2 – How to Grok an Application with Datadog<\/h1>\n\n\n\n
Whenever working with new code, it can be daunting to understand a system and how it all interacts together together.<\/p>\n\n\n\n
Our Datadog instrumentation allows us to get an immediate insight into what’s going on with the code.<\/p>\n\n\n\n
Let’s add the Datadog Agent to our docker-compose.yml<\/code>, and begin instrumenting our application:<\/p>\n\n\n
  agent:\n    image: \"datadog\/agent:6.13.0\"<\/span>\n    environment<\/span>:\n      - DD_API_KEY\n      - DD_APM_ENABLED=true<\/span>\n      - DD_LOGS_ENABLED=true<\/span>\n      - DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true<\/span>\n      - DD_PROCESS_AGENT_ENABLED=true<\/span>\n      - DD_TAGS='env:ruby-shop'<\/span>\n    ports<\/span>:\n      - \"8126:8126\"<\/span>\n    volumes<\/span>:\n      - \/var\/<\/span>run\/docker.sock:\/var\/<\/span>run\/docker.sock:ro\n      - \/proc\/<\/span>:\/host\/<\/span>proc\/:ro\n      - \/sys\/<\/span>fs\/cgroup\/:\/host\/<\/span>sys\/fs\/cgroup:ro\n    labels<\/span>:\n      com.datadoghq.ad.logs: '[{\"source\": \"datadog-agent\", \"service\": \"agent\"}]'<\/span><\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
With this, we’ve added volumes to see the resource usage on our host, along with the Docker socket so we can read the containers running on the host.<\/p>\n\n\n\n
We’ve also added a DD_API_KEY<\/code>, along with enabling logs and the process Agent. Finally, we’ve opened the port 8126<\/code>, where traces get shipped to for collection at the Agent level.<\/p>\n\n\n\n
We can now rerun our application with our DD_API_KEY<\/code> with the following command:<\/p>\n\n\n
$ export<\/span> DD_API_KEY=<YOUR_API_KEY<\/span>><\/span>\n$ POSTGRES_USER=postgres POSTGRES_PASSWORD=postgres docker-compose up<\/span><\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
And with that, we should start to see info coming in to Datadog.<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/h4>\n\n\n\n
Installing the APM Language Library<\/h2>\n\n\n\n
For Ruby on Rails, we need to first add the ddtrace<\/code> Gem to our Gemfile. Take a look at store-frontend\/Gemfile<\/code> in the Katacoda file explorer, and notice we’ve added the Gem so we can start shipping traces.<\/p>\n\n\n\n
Because we plan on also consuming logs from Rails and correlating them with traces, we’ve also added logging-rails<\/code> and lograge<\/code>. Both of these are documented on the Ruby trace \/ logs<\/a> correlation part of the documentation.<\/p>\n\n\n\n
Once these are both added to the list of our application’s requirements, we must then add a datadog.rb<\/code> to the list of initializers.<\/p>\n\n\n\n
You’ll find the file in store-frontend\/config\/initializers\/<\/code>.<\/p>\n\n\n\n
There, we control a few settings:<\/p>\n\n\n
Datadog.configure do<\/span> |c|\n  # This will activate auto-instrumentation for Rails<\/span>\n  c.use<\/span> :rails<\/span>, {'analytics_enabled<\/span>': true<\/span>, 'service_name<\/span>': 'store<\/span>-frontend<\/span>'}\n  # Make<\/span> sure<\/span> requests<\/span> \nare<\/span> also<\/span> instrumented<\/span>\n  c<\/span>.use<\/span> :http<\/span>, {'analytics_enabled<\/span>': true<\/span>, 'service_name<\/span>': 'store<\/span>-frontend<\/span>'}\n  c<\/span>.tracer<\/span> hostname<\/span>: 'agent<\/span>'\nend<\/span><\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
We set analytics_enabled<\/code> to be true<\/code> for both our Rails auto instrumentation, and the http<\/code> instrumentation.<\/p>\n\n\n\n
This allows us to use Trace Search and Analytics from within Datadog.<\/p>\n\n\n\n
We then set a hostname<\/code> for all our traces to be sent to. Because we set the Datadog Agent to listen on port 8126<\/code>, we set this to be the hostname available within our docker-compose<\/code>.<\/p>\n\n\n\n
Finally, we set an environment for our traces. This allows us to separate different environments, for example, staging and production.<\/p>\n\n\n\n
With this, our Ruby application is instrumented. We’re also able to continue traces downstream, utilizing Distributed Traces.<\/p>\n\n\n\n
Shipping Logs Correlated with Traces<\/h2>\n\n\n\n
To ship logs to Datadog, we’ve got to ensure they’re converted to JSON format. This allows for filtering by specific parameters within Datadog.<\/p>\n\n\n\n
Within our config\/development.rb<\/code>, we see the specific code to ship our logs along with the correlating traces:<\/p>\n\n\n
  config.lograge.custom_options = lambda do<\/span> |event|\n    # Retrieves trace information for current thread<\/span>\n    correlation = Datadog.tracer.active_correlation\n\n    {\n      # Adds IDs as tags to log output<\/span>\n      :dd => {\n        :trace_id => correlation.trace_id,\n        :span_id => correlation.span_id\n      },\n      :ddsource => [\"ruby\"<\/span>],\n      :params => event.payload[:params].reject { |k| %w(controller action).include<\/span>? k }\n    }\n  end<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
Next, let’s look at how a Python application is instrumented.<\/p>\n\n\n\n
<\/h4>\n\n\n\n
Instrumenting the Advertisements Service<\/h2>\n\n\n\n
In our docker-compose.yml<\/code> there’s a command to bring up our Flask server. If we look, we’ll see it’s a:<\/p>\n\n\n
flask run --port=5002 --host=0.0.0.0<\/code><\/span><\/pre>\n\n\n
Once we install the Python ddtrace<\/code> by adding it to our requirements.txt<\/code> (it should already be there), we edit this command by putting a ddtrace-run<\/code> in front:<\/p>\n\n\n
ddtrace-run flask run --port=5002 --host=0.0.0.0<\/code><\/span><\/pre>\n\n\n
With this, we’re ready to configure out application’s insturmentation.<\/p>\n\n\n\n
Automatic instrumentation is done via environment variables in our docker-compose.yml<\/code>:<\/p>\n\n\n
      - DATADOG_SERVICE_NAME=advertisements-service\n      - DATADOG_TRACE_AGENT_HOSTNAME=agent\n      - DD_LOGS_INJECTION=true<\/span>\n      - DD_ANALYTICS_ENABLED=true<\/span><\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
With this, we’ve connected and instrumented all of our services to APM.<\/p>\n\n\n\n
The last thing we need to add is a label<\/em> to our container, so our logs are shipped with the label of the service, and with the proper language processor:<\/p>\n\n\n
    labels:\n      com.datadoghq.ad.logs: '[{\"source\": \"python\", \"service\": \"ads-service\"}]'<\/span><\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
We can repeat the process, and fill out the settings for the discounts-service<\/code>:<\/p>\n\n\n
  discounts:\n    environment:\n      - FLASK_APP=discounts.py\n      - FLASK_DEBUG=1<\/span>\n      - POSTGRES_PASSWORD\n      - POSTGRES_USER\n      - DATADOG_SERVICE_NAME=discounts-service\n      - DATADOG_TRACE_AGENT_HOSTNAME=agent\n      - DD_LOGS_INJECTION=true<\/span>\n      - DD_ANALYTICS_ENABLED=true<\/span>\n    image<\/span>: \"burningion\/ecommerce-spree-discounts:latest\"<\/span>\n    command<\/span>: ddtrace-run flask run --port=5001<\/span> --host=0.0<\/span>.0<\/span>.0<\/span>\n    ports<\/span>:\n      - \"5001:5001\"<\/span>\n    volumes<\/span>:\n      - \".\/discounts-service:\/app\"<\/span>\n    depends_on<\/span>:\n      - agent\n      - db\n    labels<\/span>:\n      com.datadoghq.ad.logs: '[{\"source\": \"python\", \"service\": \"discounts-service\"}]'<\/span><\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Next, let’s take a closer look at\u00a0why<\/em>\u00a0and\u00a0where<\/em>\u00a0our application may be failing.<\/p>\n\n\n\n
<\/h4>\n\n\n\n
Step 5 – Debugging Our Application with APM<\/h1>\n\n\n\n
Now that we’ve instrumented all of our code, let’s spin up some traffic so we can get a better look at what may be happening.<\/p>\n\n\n\n
Spinning up Traffic for Our Site<\/h2>\n\n\n\n
In our \/ecommerce-observability<\/code> folder, we’ve got a copy of GoReplay<\/a>.<\/p>\n\n\n\n
We’ve also got a capture of traffic using GoReplay. Let’s spin up an infinite loop of that traffic:<\/p>\n\n\n
$ .\/gor --input-file-loop --input-file requests_0.gor --output-http \"http:\/\/localhost:3000\"<\/span><\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Once we spin up that traffic with our included observability, we can now take a look at the issues we’ve come across since the new team rolled out their first microservice, the advertisements-service<\/code>.<\/p>\n\n\n\n
Before we began instrumenting with Datadog, there’d been reports that the new advertisements-service<\/code> broke the website. With the new deployment on staging, the frontend<\/code> team has blamed the ads-service<\/code> team, and the advertisements-service<\/code> team has blamed the ops team.<\/p>\n\n\n\n
Now that we’ve got Datadog and APM instrumented in our code, let’s see what’s really been breaking our application.<\/p>\n\n\n\n
Debugging an Application with Datadog<\/h2>\n\n\n\n
The first place we can check is the Service Map, to get an idea for our current infrastructure and microservice dependencies.<\/p>\n\n\n\n
\"Datadog<\/figure>\n\n\n\n
In doing so, we can tell that we’ve got two microservices that our frontend calls, a discounts-service<\/code>, along with an advertisements-service<\/code>.<\/p>\n\n\n\n
If we click in to view our Service Overview in Datadog, we can see that our API itself isn’t throwing any errors. The errors must be happening on the frontend.<\/p>\n\n\n\n
\"Services<\/figure>\n\n\n\n
So let’s take a look at the frontend service, and see if we can find the spot where things are breaking.<\/p>\n\n\n\n
If we look into the service, we can see that it’s been laid out by views. There’s at least one view that seems to only give errors. Let’s click into that view and see if a trace from that view can tell us what’s going on.<\/p>\n\n\n\n
\"Problematic<\/figure>\n\n\n\n
It seems the problem happens in a template. Let’s get rid of that part of the template so we can get the site back up and running while figuring out what happened.<\/p>\n\n\n\n
Open store-frontend\/app\/views\/spree\/layouts\/spree_application.html.erb<\/code> and delete the line under <div class=\"container\"><\/code>. It should begin with a <br \/><\/code> and end with a <\/center><\/code>.<\/p>\n\n\n\n
The banner ads were meant to be put under store-frontend\/app\/views\/spree\/products\/show.html.erb<\/code> and store-frontend\/app\/views\/spree\/home\/index.html.erb<\/code>.<\/p>\n\n\n\n
For the index.html.erb<\/code>, under <div data-hook=\"homepage_products\"><\/code> add the code:<\/p>\n\n\n
<br<\/span> \/><\/span><center<\/span>><\/span><a<\/span> href<\/span>=\"<%= @ads['url'] %>\"<\/span>><\/span><img<\/span> src<\/span>=\"data:image\/png;base64,<%= @ads['base64'] %>\"<\/span> \/><\/span><\/a<\/span>><\/span><\/center<\/span>><\/span><\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
And for the show.html.erb<\/code> at the very bottom add:<\/p>\n\n\n
<br<\/span> \/><\/span><center<\/span>><\/span><a<\/span> href<\/span>=\"<%= @ads['url'] %>\"<\/span>><\/span><img<\/span> src<\/span>=\"data:image\/png;base64,<%= @ads['base64'] %>\"<\/span> \/><\/span><\/a<\/span>><\/span><\/center<\/span>><\/span><br<\/span> \/><\/span><\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
With that, our project should be up and running. Let’s see if there’s anything else going on.<\/p>\n\n\n\n
<\/h4>\n\n\n\n
Step 6 – Spotting and Resolving Bottlenecks with the Service List<\/h1>\n\n\n\n
With the Service List, we can see at a quick glance see endpoints that are running slower than the rest.<\/p>\n\n\n\n
If we look at the Frontend Service, we can see there are two endpoints in particular that are substantially slower than the rest.<\/p>\n\n\n\n
\"Slow<\/figure>\n\n\n\n
Both the HomeController#index<\/code> and the ProductController#show<\/code> enpoints are showing much<\/em> longer latency times. If we click in, and view a trace, we’ll see that we’ve got downstream microservices taking up a substantial portion of our load time.<\/p>\n\n\n\n
Use the span list to see where it may be, and we can then take a look at each of the downstream services and where things may be going wrong.<\/p>\n\n\n\n
It seems two microservices in particular are being called for the homepage. If we look into our docker-compose.yml<\/code>, we can see both the advertisements-service<\/code> and discounts-service<\/code> are each taking over 2.5 seconds for each request. Let’s look within their specific urls to see if there isn’t something amiss.<\/p>\n\n\n\n
Looking at the code, it appears we’ve accidentally left a line in from testing what happens if latency goes up.<\/p>\n\n\n\n
Try spotting the line and removing the code to see if you can bring the latency down again for the application.<\/p>\n\n\n\n
What sort of an improvement in page load time did it give you? Can you graph the differences over time?<\/p>\n\n\n\n
Reference<\/p>\n\n\n\n
  • https:\/\/www.katacoda.com\/burningion\/scenarios\/ecommerce-observability<\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"
    Step 1 – Spinning Up Our Legacy E-Commerce Shop Our legacy monolith shop uses Ruby on Rails and Spree. We’ve started to build out a first set of microservices, and these have been added to an initial set of containres. We use docker-compose to bring it up and running. There’s a prebuilt Rails Docker container image, along…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"_kad_post_classname":"","_joinchat":[],"footnotes":""},"categories":[2],"tags":[],"class_list":["post-30309","post","type-post","status-publish","format-standard","hentry","category-uncategorised"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/30309","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=30309"}],"version-history":[{"count":1,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/30309\/revisions"}],"predecessor-version":[{"id":30310,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/30309\/revisions\/30310"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=30309"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=30309"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=30309"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
Step 4 – APM Automatic Instrumentation with Python<\/h1>\n\n\n\n
Now that we’ve set up our Ruby on Rails application, we can now instrument our downstream Python services.<\/p>\n\n\n\n
Looking at the documentation<\/a> for the Python tracer, we have a utility called ddtrace-run<\/code>.<\/p>\n\n\n\n
Wrapping our Python executable in a ddtrace-run<\/code> allows us to spin up a running instance of our application fully instrumented with our tracer.<\/p>\n\n\n\n
For supported applications like Flask, ddtrace-run<\/code> dramatically simplifies the process of instrumentation.<\/p>\n\n\n\n
Step 3 – APM Automatic Instrumentation with Rails<\/h1>\n\n\n\n
Our code has already been set up with instrumentation from Datadog.<\/p>\n\n\n\n
Depending on the language your application runs in, you may have a different process for instrumenting your code. It’s best to look at the documentation<\/a> for your specific language.<\/p>\n\n\n\n
In our case, our applications run on Ruby on Rails<\/a> and Python’s Flask<\/a>.<\/p>\n\n\n\n
We’ll instrument each language differently.<\/p>\n\n\n\n