Today I learned how to fix a PostgreSQL error like this:
ERROR: canceling statement due to conflict with recovery DETAIL: User query might have needed to see row versions that must be removed. It’s about database replication. I have been getting that on read-only database which uses PostgreSQL Hot Standby. This error didn’t happen always but happened occasionally.
I fixed that by setting both max_standby_archive_delay and max_standby_streaming_delay to a longer time (300s) on the standby servers. The reason that this error occurs is query conflicts. This kind of error is not inevitable because of the nature of database replication, and in some cases, queries running on standby servers have to be canceled. Then the error shows up.
Today I learned that Rails 6.1.3 does not have methods that raise exceptions when more than one record was found. However, in future versions, we could use ActiveRecord::FinderMethods#sole for that purpose. It seems Enumerable#sole is also going to be introduced.
Sometimes I need to make sure there is one and only one record that matches a condition without using a unique-index, for example, when I cannot add that constraint to the database.
Today I learned/remembered that to format a number with a delimiter, we can use ActiveSupport::NumericWithFormat#to_s(:delimited).
require 'active_support' require 'active_support/core_ext' 123456789.to_s(:delimited) # => "123,456,789" Not only that, but this method also provides other formats and takes options to tweak its behavior.
123456789.to_s(:delimited, delimiter: '-') # => "123-456-789" 123456789.to_s(:currency, precision: 3) # => "$123,456,789.000" 123456789.to_s(:human_size) # => "118 MB" 123456789.to_s(:human) # => "123 Million" Originally, when I need to format a number to a delimited number, I would try to use number_to_delimited provided by ActiveSupport::NumberHelper. But to_s is handy when we have to do that where the helper is not reachable by default, such as in a serializer class for ActiveModel Serializer. Calling this method passing a format is the same as calling ActiveSupport::NumberHelper.number_to_#{format}.
In software development, a feedback loop that takes time and effort could be a major bottleneck. Especially if we have to run things on remote servers and figuring out what’s going on requires lots of effort, debugging could make frustrations. Today, I’m going to introduce a couple of tools that help to debug GitHub Actions workflows from local machines.
Act With act command, you can run workflows on your local machine. Once you run act command it reads YAML files under .github/workflows and runs the jobs in Docker containers immediately. We can make sure if there’s any major problem in the YAML without making commit and pushing to GitHub. Although act does not provide 100% compatible environment with GitHub Actions, it helps a lot when you start writing workflow settings. For example, it is useful to find any errors and mistakes in the YAML file structure, dependencies between jobs, calling actions, and steps written in shell scripts.
When we have to process a large number of files, we should be aware of the resources the process consumes. Here’s the scenario I faced in my work recently: we have to create one archive file from lots of files that are stored on an S3 bucket. Created archive files must be placed on another S3 bucket. The number of files of an archive could be 700,000. The size of each file is up to 10MiB. The size of an archive file will be less than 140GiB.
