QA » openQA Project

I'm +1 to getting this one fixed, It's quite a pain, specially when someone is trying to come up with a docker image and forgets about the part of the asset uploading :)

For the record, I recently talked to coolo about this because I'm looking into the viability of hosting workers "further away" in network terms from the server (i.e. In The Cloud). This setup is a bit of a problem for that, in a few ways.

It'd be much easier to do this in general of course if you didn't have to worry about setting up the shared filesystem on worker hosts at all. Needing it at all means you have to figure out a way to make it fly when you can't just stick up an unauthenticated NFS share since all the systems are on a private network anyway. It can be made to go with sshfs, at least according to my preliminary tests, but it's extra setup work and obviously potential fragility. I'm not sure if this is the only remaining case where the worker process may use the shared filesystem or if there are others (please point me at any others you know about!), but it's certainly one of them.

Aside from that, specifically in this case it effectively doubles the network traffic required for an asset transfer, because the worker uploads the asset to the server and then checksums the uploaded file, which effectively means it re-downloads the file via whatever the shared filesystem protocol is (assuming the server is hosting the share). So if you're uploading a 2GB asset you do 2GB of transfer from the worker to the server, then 2GB of transfer back from the server to the worker. Which is obviously inefficient and, depending on the scenario, potentially expensive.

The problem of the server potentially getting overloaded with checksum work is a real one, though, and I hadn't thought about it till coolo pointed it out...

I think this is almost the only case remaining of NFS, but it would have to be a generic checksum check for most files, and/or add verification of the upload in progress of some sort.

The worker can simply calculate the checksum and send it to the webUI along with the progress.

but who verifies the upload? we can of course also validate it on download into the cache - by every worker.

And no, it's not the only case. we also have use cases of 'other' assets delivered from share/openqa - and use them e.g. in autoyast installations. But we can find a way without them later.

I sorta felt like there must be some sort of standard way to do a 'verified' upload over HTTP (where both ends check chunks as they are transferred, or something like that), but at least from a cursory poke around Teh Intarwebz this morning, I couldn't find one.

Are 'other' assets not cached, on workers with caching enabled?

BTW, the other big roadblock I came up on was tests/needles; yes, there is caching for these, but it requires rsync (i.e. ssh), which is effectively the same as requiring a shared filesystem if you were going to use sshfs for the shared filesystem anyway. You can punch the necessary firewall holes and so on for a 'remote' worker, but it's a pain point (how much of one depends on the details of your setup, I guess).

A goofy solution I thought of for this would be to use gitfs instead of a 'traditional' shared filesystem for the distri, but a 'proper' solution might be to make it possible to define the DISTRI as a specific commit in a git repository; worker hosts would cache distri repos as they came across them, and use a temporarily-stored 'git archive' of the specified commit for each job, or something like that.

@coolo yeah, I keep forgetting about the 'others' assets lol... anywho:

but who verifies the upload? we can of course also validate it on download into the cache - by every worker.

The webUI. We can split a large file upload in small chunks configurable (say 10MB/100MB), worker calculates a checksum for said chunk adding a header to the transaction, Once worker marks a chunk as the last one, webUI verifies checksum, tells the worker and everybody is happy...

This could allow to save time in case of network errors.

Crazy idea, but who knows!

This is doing something like that: https://gist.github.com/jberger/4744482

szarate wrote:

The webUI. We can split a large file upload in small chunks configurable (say 10MB/100MB), worker calculates a checksum for said chunk adding a header to the transaction, Once worker marks a chunk as the last one, webUI verifies checksum, tells the worker and everybody is happy...

I wonder if this provides any benefit to letting the webui just calculate the checksum over the whole large file whenever it's completed. Leave the chunking responsibility to an efficient checksumming algorithm rather than trying to roll your own.

Yeah, because mojolicious is nonblocking - your requests are supposed to be easy to digest.