tickets #133106
openlogging endpoint for fastly CDN
0%
Description
Hi all,
currently opensuse.org is operating a metrics analytics under metrics.opensuse.org which iirc is somehow fed from the apache logfiles of download.opensuse.org. if we switch to a CDN, we would lose these insights and hence I think it is useful to start collecting the logs from the CDN instead.
from the list of integrations available at https://docs.fastly.com/en/guides/integrations#_logging-endpoints which one do you prefer? to me the https one seems the most obvious one, but it has a custom json format that probably needs to be converted by a receiver into a line based apache-like format.
goal of this ticket is to finish log collection so that metrics.o.org is gathering the right amount of data.
Updated by crameleon 10 months ago
- Private changed from Yes to No
Hi Dirk,
the logs could possibly be injected into our Minio based S3 server, if their "Amazon S3" feature allows for a custom endpoint to be configured - I think ours is currently not exposed to the internet, but that could be changed. Would that be easier than crafting some custom HTTP endpoint? From the S3 bucket the logs could then be consumed by one or more other internal services to generate statistics.
I'm not sure if this is redundant to the collection of the logs, but would it make sense to run https://github.com/fastly/fastly-exporter and feed it to our Prometheus/Grafana setup?
Cheers,
Georg
Updated by dirkmueller 10 months ago
It appears that it is possible to specify a custom domain in the Amazon S3, so that could be working. I'm not certain that we can specify a custom port however. worth a try. the alternative is of cource https posts -> forwarded then to whatever metrics collection we want to use.
fastly-exporter is a way to create beautiful dashboards, but it will not help us with usage tracking as the necessary headers etc are not exposed there as far as I know. it does help to troubleshoot cdn however.