Not sure if it will be useful but if you are running a physical server, I occasionally have access to used CCTV Servers. Mostly Dell poweredge.
Sometimes upwards of 16 terabyte raided at raid 5 on storage (occasionally independently running raid1 for O/s).
It’s a virtual server. Yes indeed, if you do the other way (have your own PC sitting at a hosting centre) then storage is basically free. But virtual servers are sold per GB and for us this is the best solution. I was able to “fix” this from home. I have auto backups and loads of other services. I “inherited” the whole project 4 years ago and there has been zero downtime.
I used to run the online shop + website at work with self-hosting. A nightmare, with a backup server off-site (i.e. in a downstairs toilet at home) and all kinds of stuff to make it all actually usable. Depended totally on the part-time IT guy we had (10k/year and this was a part of his job). Nothing documented (of course). Now it is virtually hosted too and just keeps going. Yeah… limited storage but OK.
same same on the captcha…
I think, in the long run, I will need someone familiar with
and do it over Zoom, so no need to hand out logins, etc.
I think the server is NGINX.
Did not get a captcha, only the Cloudflare ‘security check’.
Regarding database disk usage: check if it is binary logs that fills up the disk
The captcha has been set to a low level.
All websites get bots and crawlers and those with some “data” also get scrapers. So this is normal. But this one is a bit different in that it runs queries which generate large amounts of data which causes logfiles to grow fast. 0.5MB to 1MB bytes per minute. The details are above me unfortunately but it will get solved.
It may also be a consequence of something else. Recently one pilot had a problem with our airport database. Upon investigation I found it was getting loads of hits from China and Russia. Nothing to swell up storage; just loads of pointless hits, also collecting tafs and metars for random airports (which is not good because that generates traffic elsewhere). So I blocked all traffic from China and Russia for several websites I look after. China has zero GA, and we have not had a Russian post on GA matters for about 8 years. Well, we sort of have but they were pretending to be not in Russia 
And it is quite possible that some Russian got p1ssed off, especially as our “Russian invasion” thread has been less than fully supportive of Putin’s latest foreign policy experiment.
Peter_Mundy wrote:
Ditto – and done
Same here. Good luck with this. I’m afraid I can’t help much on IT…
Does it look more like “incompetence” than “malice”? (In other words, a crawler that’s perhaps not parsing robots.txt but identifies itself with a useful user agent string nonetheless rather than masquerading as a desktop browser) – you may just be able to filter out certain things based on user agent. Nginx has several options for this, it’s quite configurable (although it’s been a while since I’ve worked on any of that so I can’t give any immediate advice – but the nginx docs are a good place to start reading because you need to control this stuff at the HTTP level more than the packet filtering level).
