So you've got a content-oriented website, maybe your own blog or something, and maybe you've (like me) decided to ignore the advice about using static site generators. You build your Rails site, and it is wonderful and beautiful and dynamic, and every page it replies with delights readers with your artisanally-crafted HTML. Maybe you've got some internal caching (Rails has you covered here), maybe it's all roundtrips to the database. But who cares! Your site is up and receiving traffic!
Then, suddenly, a storm hits. Congratulations! You've made it to the top of Reddit/Slashdot/Hacker News! You now have thousands, if not millions, of people beating down your door to read your content. But now your site is down! The link's comment thread is filling up with complaints about being hugged to death, and a few helpful souls are posting the archive.org equivalent of your link and siphoning away your traffic.
How do we fix this?
You could throw more compute resources at it -- think "scaling vertically/horizontally" -- which I'm sure your server/application host would ab$olutely love.
Or, you could install some sort of proxy cache in front of it. The traditional answer here is to use nginx or Varnish as a caching proxy, but if you use a content delivery network (such as Cloudflare) it may be better to use that CDN's caching features instead. (Some might recommend using both your own cache and your CDN's cache, but I wouldn't advise this because troubleshooting cache issues is already difficult enough, and having multiple layers only makes debugging even more confusing. If you do this, you should understand your web application thoroughly.)
Since this site is fronted by Cloudflare, I want to make use of its page cache: it's free and comes with the service!
However, setting this up is not as simple as it may first appear: in a default configuration, Rails doesn't permit caching (the Cache-Control headers it sends don't allow for it), and as a result, nearly every request you receive bypasses the cache and gets passed directly to the app server. This is a screenshot of my Cloudflare dashboard showing the percentage of page requests cached before I applied the fixes I describe here (those peaks top out at ~10%):
Uh.... that's not very good!
Now, you can set up rules in the Cloudflare dashboard to override Rails' requested caching behavior, but this does not solve the underlying root cause: Rails is requesting no caching, because the Cache-Control request header it sets explicitly forbids it:
Cache-Control: NO CACHE!
Setting the Correct Cache-Control Headers with Rails
NOTE: The directions given here apply to Ruby on Rails version 7, though expires_in and fresh_when have existed since at least version 2, and concerns have been available since version 4.
Luckily, Rails makes changing this behavior fairly simple. We don't even need to really dive into how Cache-Control works! (Though here is a good guide if you want to know.) You simply call the expires_in and/or fresh_when functions in your controller, supplying an expiry and ensuring that you set public to true. Like this:
expires_in 1.hour, public: true
# or
fresh_when(@article, public: true)
However, setting this for every route is both tedious and a pretty egregious violation of DRY. Instead, we can set as much as we can once and then propagate it through our application using either class inheritance or composition (via ActiveSupport's concerns) feature. And while inheritance may be slightly easier, composition is a bit more modern and flexible; here we will be taking the latter approach.
To start, we will want to make a new concern and call it "Cacheable". The easiest way to do this is to simply go to the $RAILS_ROOT/app/controllers/concerns folder and create a new file, naming it cacheable.rb. In this file, we want to make one small action (called "set_cache_headers") and call expires_in within it. Here is a very basic and usable example, which also prevents page caching when a user is logged in:
# app/controllers/concerns/cacheable.rb
module Cacheable
extend ActiveSupport::Concern
included do
before_action :set_cache_headers, unless: current_user
end
def set_cache_headers
expires_in 1.hour, public: true
end
end
Then, for each controller whose content you wish to cache, simply add "include Cacheable" at the top right below the class declaration. Here is an example pulled directly from this site's code, for the controller that powers the "About" feature:
# app/controllers/static_controller.rb
class StaticController < ApplicationController
include Cacheable
def about
@page_title = "About"
end
end
Once this is done you will see that Cache-Control is indeed being set correctly:
Objective achieved!
But! You are not finished yet! You may notice that while your 'cached' stats are going up, they aren't going up as much as one might think. This is because there is another component to page caching that we have not yet discussed: etags.
Setting the Correct Etag Headers with Rails
This is where things get a bit more tricky: Rails generates another header, called an Etag, that, in theory, is supposed to be unique for each page. (For the more technically inclined, you can think of an Etag as like a SHA256 hash for your page.) But Rails, by default, makes this tag unique per request. Both your browser cache and your CDN cache read this header to determine whether a given request is a cache hit or cache miss, and so we will need to configure Rails' to set it correctly, based on our rendered page content (or other context).
Enterfresh_when, which provides further direction to Rails on how to render the correct etag header. You provide it with an object that describes what the page renders -- generally the model instance for the given page (the Rails docs use @article in their examples) -- and it generates a hash that is used for the Etag header.
Using fresh_when with Dynamic Pages
For dynamic pages, such as the example described above and in the Rails docs, simply call fresh_when and pass the it your model instance as its first parameter, inside the controller route. Like so:
# app/controllers/articles_controller.rb
def show_article
fresh_when @article
end
When combined with aforementioned Cacheable, this is sufficient to avoid page-caching in the case of logged-in users, as the expires_in directive is never called when current_user exists, and Rails reverts to its default, zero-expiry "private" cache behavior.
If you aren't using Cacheable, as described above, you will need to consult the documentation as you need to provide additional information.
Using fresh_when with Static Pages
Static pages are a bit more tricky here, as the examples in the docs do not cover this circumstance. Once again by default the etag will always be different, so we need to pin it to something, yet appropriate data (namely, the path to the view and a timestamp for when it was last updated) isn't really available to our controller.And, like above, we don't want it to get mixed-up when the site is being viewed by a logged-in user vs. the anonymous public.
I haven't figured out a great solution here, but we can build a string using params and current_user and set that as the value of our etag, and it should work well enough for our purposes. But! You will need to manually purge the cache on your CDN when these pages are updated, or wait for them to expire. For this case, a short expiry (say, 15 minutes) is useful here.
(Note that if we can get information about the last-modified timestamp of the template to be rendered, we could include that data in the etag so it would naturally invalidate when it is updated, but I don't know of a non-hacky way to do this and some preliminary research in this area yields nothing.)
So, we craft another concern, this time called StaticCacheable, and include this in each controller serving static content. Once again, like Cacheable, this is a controller-level solution; if you need something per-action that is an exercise left up to you.
To make this concern, create a new file called static_cacheable.rb and save it to your $RAILS_ROOT/app/controllers/concerns folder, right next to cacheable.rb. Note that we will include a reference to Cacheable from within StaticCacheable, so that you only need to include StaticCacheable on your static controllers. In the action it defined we simply grab route information from params and feed that into fresh_when:
# app/controllers/concerns/static_cacheable.rb
module StaticCacheable
extend ActiveSupport::Concern
included do
include Cacheable
before_action :set_static_cache_headers
end
def set_static_cache_headers
etag = "#{params[:static]}#{params[:about]}#{current_user}"
fresh_when etag: etag
end
end
Note that including current_user makes the Etag different every for request, when current_user is logged in. This is because fresh_when will coerce the string representation of current_user (via an implicit .to_s), which will always vary because that string includes current_user's internal object-id (NOT its database id), which varies with each request.
Finally!
Once your Cache-Control and Etag headers are under control and correctly set, and you have correctly configured your proxy service, your site should be well-equipped to handle large volumes of traffic without falling over. Hurray!
A Quick Note About Cloudflare and Implementing This
It's worth noting that Cloudflare seems to strip the Etag header when Cache-Control renders it useless, as is the case when Cache-Control is set to private. This may seem annoying but it punches out your browser cache, presumably to ease troubleshooting.
You can still see the Etag header if you pull requests directly from your webserver (by its IP address), and it will also be visible during development. Unfortunately, it seems like you will have to rely on unit tests or Cloudflare's provided statistics to verify your cache strategy is working.
If you are writing a Windows-only or cross-platform application using pyQT or pySide (version 6, as of this time of writing), you may discover that you need to change your application's icon, as it appears in the taskbar. By default, it is a boring, generic "window" icon, and, naturally, you will want to change it!
But the instructions for doing so aren't very clear. This StackOverflow post gets us started, but the given (and accepted) solutions seem to advise the unnecessary extra step of saving a copy of the icon as a Python variable itself!
There is a simpler way, and it involves just a few lines of code. Inside the __init__() function of your QT window's code, add the following:
from pathlib import Path
from PySide6.QtGui import QPixmap, QIcon #substitute the equivalent pyQT line if you're using pyQT
# load window icon
path_to_icon = 'res/icon.png' #the path to your icon file, relative to the project root
pixmap = QPixmap()
pixmap.loadFromData( Path( path_to_icon ).read_bytes() )
appIcon = QIcon(pixmap)
self.setWindowIcon(appIcon)
And voilà! You now have a pretty icon in the taskbar for your program.
There's a new "Photography" section on this site, and I want to highlight something I've been doing for years that has kind of flown under the radar: for a while, I maintained some galleries on Flickr of my photography formatted for display as wallpapers for your devices. It was cool, but it kind of languished, lost in a sea of public Flickr galleries that few ever saw.
So, I decided to integrate those galleries with this website. Now, with this site and blog in full swing, I can bring a bit more attention to it! Plus, the images have been recut and polished just a wee bit more!
If you use Bulma CSS, Turbo, and Stimulus in your web project, you may have noticed that the hamburger menu in the navbar doesn't work, and that you need to supply your own Javascript to make it work. While this is explained in the docs -- Bulma doesn't include any Javascript, after all -- the plain JS example provided doesn't work on websites using Turbo.
This occurs because, on a new page load, Turbo replaces the click handler's DOM element with new elements, even if that aspect of the page hasn't changed, without calling DOMContentLoaded (or any other startup event). Meaning, we cannot use DOMContentLoaded to set our click handlers!
So we need something else: instead of DOMContentLoaded we can use Stimulus to add a little "hamburger helper" to the project and get that menu working!
First, we want to create a new Stimulus controller, which we will call navbar_controller.js. In it, we will create one action, toggleMenu(), which will handle the click event and control the hamburger menu by toggling the is-active class on the hamburger HTML elements. In that file, add the following code:
import {Controller} from '@hotwired/stimulus';
// Navbar toggle
// Adapted from https://bulma.io/documentation/components/navbar/#navbar-menu
export default class extends Controller {
toggleMenu() {
// Get all "navbar-burger" elements
const $navbarBurgers = Array.prototype.slice.call(document.querySelectorAll('.navbar-burger'), 0);
// Get the target from the "data-target" attribute
const target = this.element.dataset.target;
const $target = document.getElementById(target);
// Toggle the "is-active" class on both the "navbar-burger" and the "navbar-menu"
this.element.classList.toggle('is-active');
$target.classList.toggle('is-active');
}
}
Then, in your navbar HTML code, find the <a role="button" class="navbar-burger"> tag that comprises the hamburger button and add the following data attributes:
data-controller: set this to the name of your Navbar controller: "navbar"
data-action: use this to bind the click event to toggleMenu(): "click->navbar#toggleMenu"
data-target: set this to the id of your navbar <div> element, which should have a class of navbar-menu. On this site, the element's id is "ze-navbar", so that would be the value for this data attribute.
Once you've done that, your .navbar-burger HTML element should look something like this:
And that should be it! Reload the page and everything should work nicely.
I know this seems extremely basic, but if you are like me and still a bit overwhelmed by all the quirks and eccentricities of building a web app in Turbo, you might be thankful to have it all spelled out for you. I certainly would have liked that!
One of my favorite computer games over the last few years has been X4: Foundations, a spaceflight-sandbox-economy-eurojank-sim 4X game made by the fine folks at Egosoft. Starting a new game in X4 can be an overwhelming experience, as you have no real objective, no starting cash, no purpose, and are generally unable to effect the world the game builds around you. (All this will flip around, with persistence...)
So, how then to get started? And how to turn that into an empire? There are many options, but for this post I will be discussing one of them. It is not the fastest nor easiest way to make space bucks from the start, and it can be a little tricky to get running, but if you stick with it, you can build a consistent passive income for your empire that runs in the background while you play.
At present this strategy is generating roughly 20M credits per hour for me.
So how does it work? That's easy! We corner the market for silicon. As the most popular of the four solids in the game, silicon can experience extremely high demand in certain sections of the game map -- namely, Terran space. And, unlike Ore, a full shipment of silicon sells for a decent amount of cash.
Terrans (and the Segaris Pioneers) are fantastic for this strategy because their economy and build requirements are very streamlined compared to the other species in X4 -- they have just three manufacturing goods: Silicon Carbide, Metallic Microlattice, and Computronic Substrate. As silicon is needed for two of those three, and because of the reduced bill of materials for pretty much everything in their economy, they need to build a ton more of these seemingly basic goods. Which means a near-insatiable demand for silicon can be found in Terran space, which the stock gameworld underserves to a near criminal degree!
The entire Terran tech tree...
If Terran space isn't to your liking, another place with a high demand for silicon is Grand Exchange, which is even better if you don't have the Terran expansion pack. However, this area has a strong pirate presence and a high chance of Kha'ak harassment, so it needs a bit more babysitting and investment to get running to the point of being entirely hands-off.
Another reason I like the Terrans is that their mining ships are cheap and expendable, and the S-class ships have comparatively high cargo space compared to the other species. One Kopis (the Terran S miner) has two-thirds the cargo space of the Bolo (the Terran M miner), but costs less than half of one! Terran space is also relatively safe -- the only thing you need to worry about are occasionally Kha'ak stations spawning nearby and harassing your miners. (Which you will need to deal with at some point, but can be worked around.)
The foundation (heh...) of this strategy is what I call the Solids Distribution Hub, which is a simple station serving as headquarters for your silicon miners and traders. You will need a manager (the more stars the better!), and a roughly equal number of identical ships mining and trading for the station. (Or, if you want to use S-class miners like Kopises (Kopii?) and Bolos, roughly a 3:2 ratio of S-class miners to M-class traders.)
Prerequisites
This strategy will require a fair amount of credits to get started, as you will need to buy the schematics for a dock and solids storage. You will also need to curry faction favor enough to be able to buy the necessary blueprints. (All this can be skipped if you start with cash and/or blueprints already unlocked....)
From a new game starting with 0 credits, here are some ideas for getting your seed capital:
Hunt for crystals, which is slow as crystal payouts were nerfed several patches ago, or
Search for lockboxes, claim their contents, and sell off any high-value items within, or
Shoot stuff and collect bounties
That should get you enough money to start buying cheap miners, which you can set to Sector Automine. I like to do this in Segaris or Gaian Prophecy. Let these guys do their thing until your Segaris Pioneers reputation is +10, then they will let you buy station-building blueprints. Once you have the blueprints for a dock and solids storage, you can build your first Solids Distribution Hub.
How To Do It
1. Build the Solids Distribution Hub
As I mentioned just above, the cornerstone of this strategy requires building one or more Solids Distribution Hubs. These are spacestations with nothing more than a dock, and one or more solids storage modules. (I like M size for this, but S or L will work too.)
You want to choose a location that is close to ample raw silicon (use resource probes in purple hex tiles to find out), and no more than a couple of jumps away from customers with a healthy demand for silicon. For Terrans, you want to be near to Silicon Carbide and Computronic Substrate factories, and you will want to focus on silicon. In other sectors you will want to be near Silicon and/or Ore Refineries, or Teladianium Foundries (if in Teladi space), and you will want to trade in both ore and silicon.
In Terran space, Brennan's Triumph is my favorite spot, but Gaian Prophecy and Segaris work too: they all have large silicon deposits nearby and plenty of consumers to sell to. Brennan's Triumph is the best, in my opinion, as a five-star manager will let you reach all of the outer Terran planets between Asteroid Belt to Oort Cloud. Once you have gained access to the inner Terran planets, Mars is also a good spot, as there are a number of consumers in that sector, Asteroid Belt is only a single jump away, and a five-star manager can reach all the remaining Terran planets. I also like to build one in Grand Exchange, though the subsector you choose doesn't really matter. (In previous games, I'd use the free HQ as one such hub, but with the Boron expansion its starting location has moved.)
Note that you may need to raise your reputation with the Terrans to -9 or higher in order to trade with them. You can do so by shooting criminal traffic at any TER station.
Once the space station is complete...
2. Assign a manager
Be sure to assign the most qualified manager you have, as this will greatly affect the hub's efficacy and range of sales (unless you use mods that provide better trading commands). Jump range for both miners and traders is roughly 1 jump per the higher value: manager or pilot star-rating. I find that you can oftentimes find some decent managers hiding in your organization as service crew members, and it is easy to transfer them to the role.
3. Make a new trading rule called "Mine Only" and set it as default for your hub
To do this, go to the "Global Orders" tab of your "Player Information" screen, and make a new trade rule that permits trading only with your faction. Do not set it as default for anyone. It should look like this:
My faction for this save is called the "New Paranid Empire", and this rule will force my station to trade only with my ships.
4. Set your Solids Distribution Hub to buy only from you
Unfortunately station manager's AI does not know how to handle a distribution hub such as this, so you need to do some manual setup on the station, otherwise your traders will stand around aimlessly with nothing to do. We basically need to override its automatic buy/sell behavior to always buy and sell as much as possible, and to only buy from us.
To do this, we need to manually choose our trade goods, and manually set the target buy and sell amounts to be the absolute maximum and minimum inventory levels. You can set trade goods in the Logistics view for your station, by selecting the "Silicon" (and "Ore", if you want to trade that) entries in the dropdown towards the middle-bottom of the screen.
Once your trade goods are set, open up the details panel for the good and set it to buy up to the max and sell down to the minimum. Make sure the trade rule on the buy side is set to your "Mine Only" rule. You can leave the automatic pricing or set it yourself; buying from your own miners is always free. Note that you must click "Add Sell Offer" and "Add Buy Offer" to get to these detail screens, and then uncheck the "Automatic" buy/sell box. I have highlighted the relevant sections you need to change:
Note the circled settings
5. Deploy the miners!
Once your station is configured, you are ready to start mining and trading! You can use any size miner you want; to start I prefer using the Terran S-class miners (Kopis) as they are cheap (about 250k each for a minimum spec) and Terran M-class miners (Bolo) for trading. For these guys, buy them in a 3:2 ratio of S to M. Otherwise, use M-class miners and traders and buy them in a ratio of 1:1. No matter what, you will always be buying solids miners, but you can save a bit of cash by skipping the purchase of mining hardware for the trade ships.
Also note that we are mixing S- and M-class ships for a reason: this dramatically increases your total throughput as you are making full use of available parking, which means less ships standing around waiting to dock. It would be cheaper to use only S-class ships, but they would spend more time waiting around when all available parking is full.
In my personal experience, for stations with a 5-jump radius, the local TER/PIO market seems to get saturated at around 40 S-class miners. This number is likely smaller if your station's jump range is smaller, and it will grow larger as the economy around your stations comes fully online.
You will want to balance inflows and outflows of raw materials, and ideally your station's storage is never full nor empty. If it's full and miners are complaining, add more traders; if it's empty and traders are complaining, add more miners. When the asking price for silicon begins to drop from it's max of 151, you are beginning to saturate the market. Once it has dropped to its minimum, all of your customers are at max capacity and you have fully saturated your market. At this point traders will start complaining about a lack of trades, even if your storage is full, and you will notice that profitssss from your trades have fallen greatly.
6. Watch the $$$ roll in
If the station is configured correctly, your miners will now go out and mine for the station manager when they are assigned to "Mine for this commander". Similarly, your traders need to be set to trade for the station. Pay special attention to the station's logistics configuration as this needs to be exactly right; if it is not set correctly then the miners will do nothing.
The station manager will send you cash on a regular basis, usually several times a minute as trades roll in.
Now you can sit back, do your things, and watch your in-game credit account slowly shoot upwards!
Yeehaw!
Dealing with the Kha'ak
Inevitably, the game will eventually spawn a Kha'ak station somewhere along the outer edges of your sector, and start harassing your miners. You can tell this is happening because you will start to receive notifications that miners are being attacked and/or destroyed by KHK ships.
Left unchecked, these infestations will eventually kill all of your miners. It is important to address this situation early, as you can't really rely on your allies to solve the problem for you. Luckily, they only seem to attack miners engaged in mining, and not miners simply transiting through the sector, so the easiest solution is to blacklist "sector activities" for the affected sector and cause your miners to go elsewhere. This will affect your mining throughput, however, as miners now need to travel further to search for mining spots.
Therefore, you will want to gather a team of scouts to track down the Kha'ak. (I call them... the Kha'ak Hunters!) You can do this by purchasing six fast scouts with good travel engines, such as TER Rapiers, and ordering each one of them to explore one corner of the sector's hex map borders. After a few minutes they will likely stumble upon the station, and you will have to gather up your forces to take it out!
If you are close allies with the faction that owns the sector in question, sometimes they will dispatch a force to take it out for you. In this case I'm not really sure about whether you have to scout the station out or not, so I will do the scouting run anyway.
But take note that, per this forum post from a moderator at Egosoft, the Kha'ak will stay gone much longer if you take them out (24 in-game hours!). If one of the other factions does instead, they are only locked out of spawning for an hour.
In Conclusion
Silicon mining provides a solid foundation for the budding galactic emperor, and should give you enough money to purchase a defensive force and any infrastructure you need for the next stage of empire-building. It is almost completely passive, and requires very little ongoing maintenance or attention. Mining ships are cheap and expendable, and their cargoes aren't worth enough for pirates to bother with. It is a great way to build a solid income for playing the rest of the game!