The MM1 publiclist links are supposed to still work. If an old
publiclist link tries to directly access a product/version or
product/version/arch combination an additional 'mirrors' needs to be
inserted into the link. For full MM1 compatibility following PR is
necessary:
https://github.com/fedora-infra/mirrormanager2/pull/76
Since MM2 is in production about 140 mirrors have been auto-disabled
due to crawler timing out after 2 hours (default). Try if it works
better with 3 hours. This in combination with the previous commit
to decrease the load on the database should help to auto disable
less good mirrors. Especially mirrors who mirroring almost
everything can hardly be crawled within the 2 hour limit. Unfortunately
the limit is per host and not category.
Even with rsync as crawl method some hosts are taking a very long time
to be crawled. The network connection with rsync is only open for a
short time, but with both crawlers reading and writing from the database
it takes a very long time until the status of all directories is
updated. Therefore this patch introduces a 3 hour delay of the crawl
on the second crawler. This could also be solved with two different
cron.d files; one for each crawler.
The script which checks if umdl should be run used to store
the time of the end of the current umdl run. It actually should
remember the time of the start of this run as we want to make sure
to pick up all changes since the start of the current run.
Also print the date and time in the same line as the logging.
This commit adds two new scripts to the crawlers:
* mm2_get-highest-active-host-id: this queries the MM database and returns
the id of the active mirror with the highest id
* run_crawler.sh: this calculates the right startid and stopid parameters
depending on the number of available crawlers
These scripts are integrated into the start of the crawler so that each
crawler only crawls a subset of the existing mirrors. This distribution
is not perfect as it pretends that the active mirrors are equally
distributed under the total number of existing mirrors.
/usr/bin/mm2_update-master-directory-list is now wrapped with a
script which queries the fedmsg bus if there is actually new data
on the master mirror. So now it is wrapped with a lock wrapper
and fedmsg query wrapper.
to make sure the files are all there. The script is
looking in date dependent directories and depending
on the timezone of the remote server the files appear late.
