In the past, different approaches to this goals were implemented. The old NFS file system is an attempt to implement network transparency on the level of the POSIX API. While this approach works quite well in local, closely coupled networks, it does not scale for resources to which access is unreliable and possibly slow. Here, asynchronicity is important. While you are waiting for your web browser to download a page, the user interface should not block. Also, the page rendering should not begin when the page is completely available, but should updated regularly as data comes in.
In the KDE libraries, network transparency is implemented in the KIO API. The central concept of this architecture is an IO job. A job may copy, or delete files or similar things. Once a job is started, it works in the background and does not block the application. Any communication from the job back to the application - like delivering data or progress information - is done integrated with the Qt event loop.
Background operation is achieved by starting ioslaves to perform certain tasks. ioslaves are started as separate processes and are communicated with through UNIX domain sockets. In this way, no multi-threading is necessary and unstable slaves can not crash the application that uses them.
File locations are expressed by the widely used URLs. But in KDE, URLs do not only expand the range of addressable files beyond the local file system. It also goes in the opposite direction - e.g. you can browse into tar archives. This is achived by nesting URLs. For example, a file in a tar archive on a http server could have the URL
http://www-com.physik.hu-berlin.de/~bernd/article.tgz#tar:/paper.tex
void FooClass::makeDirectory() { SimpleJob *job = KIO::mkdir(KURL("file:/home/bernd/kiodir")); connect( job, SIGNAL(result(KIO::Job*)), this, SLOT(mkdirResult(KIO::Job*)) ); } void FooClass::mkdirResult(KIO::Job *job) { if (job->error()) job->showErrorDialog(); else cout << "mkdir went fine" << endl; } |
Depending on the type of the job, you may connect also to other signals.
Here is an overview over the possible functions:
The following types are currently defined:
Although the way of storing information about files in a UDSEntry is flexible and practical from the ioslave point of view, it is a mess to use for the application programmer. For example, in order to find out the MIME type of the file, you have to iterate over all atoms and test whether m_uds is UDS_MIME_TYPE. Fortunately, there is an API which is a lot easier to use: the class KFileItem.
KURL source, target; source = ...; target = ... KIO::NetAccess::copy(source, target); |
The function will return after the complete copying process has finished. Still, this method provides a progress dialog, and it makes sure that the application processes repaint events.
A particularly interesting combination of functions is download() in combination with removeTempFile(). The former downloads a file from given URL and stores it in a temporary file with a unique name. The name is stored in the second argument. If the URL is local, the file is not downloaded, and instead the second argument is set to the local file name. The function removeTempFile() deletes the file given by its argument if the file is the result of a former download. If that is not the case, it does nothing. Thus, a very easy to use way of loading files regardless of their location is the following code snippet:
KURL url; url = ...; QString tempFile; if (KIO::NetAccess::download(url, tempFile) { // load the file with the name tempFile KIO::NetAccess::removeTempFile(tempFile); } |
void FooClass::reloadPage() { KURL url("http://www.kdevelop.org/index.html"); KIO::TransferJob *job = KIO::get(url, true, false); job->addMetaData("cache", "reload"); ... } |
The same technique is used in the other direction, i.e. for communication from the slave to the application. The method Job::queryMetaData() asks for the value of the certain key delivered by the slave. For the HTTP slave, one such example is the key "modified", which contains a (stringified representation of) the date when the web page was last modified. An example how you can use this is the following:
void FooClass::printModifiedDate() { KURL url("http://developer.kde.org/documentation/kde2arch/index.html"); KIO::TransferJob *job = KIO::get(url, true, false); connect( job, SIGNAL(result(KIO::Job*)), this, SLOT(transferResult(KIO::Job*)) ); } void FooClass::transferResult(KIO::Job *job) { QString mimetype; if (job->error()) job->showErrorDialog(); else { KIO::TransferJob *transferJob = (KIO::TransferJob*) job; QString modified = transferJob->queryMetaData("modified"); cout << "Last modified: " << modified << endl; } |
Behind the curtains, the scenario is a lot more complicated. When you create a job, it is put in a queue. When the application goes back to the event loop, KIO allocates slave processes for the jobs in the queue. For the first jobs started, this is trivial: an IO slave for the appropriate protocol is started. However, after the job (like a download from an http server) has finished, it is not immediately killed. Instead, it is put in a pool of idle slaves and killed after a certain time of inactivity (current 3 minutes). If a new request for the same protocol and host arrives, the slave is reused. The obvious advantage is that for a series of jobs for the same host, the cost for creating new processes and possibly going through an authentication handshake is saved.
Of course, reusing is only possible when the existing slave has already finished its previous job. when a new request arrives while an existing slave process is still running, a new process must be started and used. In the API usage in the examples above, there are no limitation for creating new slave processes: if you start a consecutive series of downloads for 20 different files, then KIO will start 20 slave processes. This scheme of assigning slaves to jobs is called direct. It not always the most appropriate scheme, as it may need much memory and put a high load on both the client and server machines.
So there is a different way. You can schedule jobs. If you do this, only a limited number (currently 3) of slave processes for a protocol will be created. If you create more jobs than that, they are put in a queue and are processed when a slave process becomes idle. This is done as follows:
KURL url("http://developer.kde.org/documentation/kde2arch/index.html"); KIO::TransferJob *job = KIO::get(url, true, false); KIO::Scheduler::scheduleJob(job); |
A third possibility is connection oriented. For example, for the IMAP slave, it does not make any sense to start multiple processes for the same server. Only one IMAP connection at a time should be enforced. In this case, the application must explicitly deal with the notion of a slave. It has to allocate a slave for a certain connection and then assign all jobs which should go through the same connection to the same slave. This can again be easily achieved by using the KIO::Scheduler:
KURL baseUrl("imap://[email protected]"); KIO::Slave *slave = KIO::Scheduler::getConnectedSlave(baseUrl); KIO::TransferJob *job1 = KIO::get(KURL(baseUrl, "/INBOX;UID=79374")); KIO::Scheduler::assignJobToSlave(slave, job1); KIO::TransferJob *job2 = KIO::get(KURL(baseUrl, "/INBOX;UID=86793")); KIO::Scheduler::assignJobToSlave(slave, job2); ... KIO::Scheduler::disconnectSlave(slave); |
You may only disconnect the slave after all jobs assigned to it are guaranted to be finished.
protocoldir = $(kde_servicesdir) protocol_DATA = ftp.protocol EXTRA_DIST = $(mime_DATA) |
The contents of the file ftp.protocol is as follows:
[Protocol] exec=kio_ftp protocol=ftp input=none output=filesystem listing=Name,Type,Size,Date,Access,Owner,Group,Link, reading=true writing=true makedir=true deleting=true Icon=ftp |
The "protocol" entry defines for which protocol this slave is responsible. "exec" is (in contrast what you would expect naively) the name of the library that implements the slave. When the slave is supposed to start, the "kdeinit" executable is started which in turn loads this library into its address space. So in practice, you can think of the running slave as a separate process although it is implemented as library. The advantage of this mechanism is that it saves a lot of memory and reduces the time needed by the runtime linker.
The "input" and "output" lines are not used currently.
The remaining lines in the .protocol file define which abilities the slave has. In general, the features a slave must implement are much simpler than the features the KIO API provides for the application. The reason for this is that complex jobs are scheduled to a couple of subjobs. For example, in order to list a directory recursively, one job will be started for the toplevel directory. Then for each subdirectory reported back, new subjobs are started. A scheduler in KIO makes sure that not too many jobs are active at the same time. Similarly, in order to copy a file within a protocol that does not support copying directly (like the ftp: protocol), KIO can read the source file and then write the data to the destination file. For this to work, the .protocol must advertise the actions its slave supports.
Since slaves are loaded as shared libraries, but constitute standalone programs, their code framework looks a bit different from normal shared library plugins. The function which is called to start the slave is called kdemain(). This function does some initializations and then goes into an event loop and waits for requests by the application using it. This looks as follows:
extern "C" { int kdemain(int argc, char **argv); } int kdemain(int argc, char **argv) { KLocale::setMainCatalogue("kdelibs"); KInstance instance("kio_ftp"); (void) KGlobal::locale(); if (argc != 4) { fprintf(stderr, "Usage: kio_ftp protocol " "domain-socket1 domain-socket2\n"); exit(-1); } FtpSlave slave(argv[2], argv[3]); slave.dispatchLoop(); return 0; } |
Additionally, there are reimplementable functions not listed in the .protocol file. For these operations, KIO automatically determines whether they are supported or not (i.e. the default implementation returns an error).
All these implementation should end with one of two calls: If the operation was successful, they should call finished(). If an error has occured, error() should be called with an error code as first argument and a string in the second. Possible error codes are listed as enum KIO::Error. The second argument is usually the URL in question. It is used e.g. in KIO::Job::showErrorDialog() in order to parametrize the human-readable error message.
For slaves that correspond to network protocols, it might be interesting to reimplement the method SlaveBase::setHost(). This is called to tell the slave process about the host and port, and the user name and password to log in. In general, meta data set by the application can be queried by SlaveBase::metaData(). You can check for the existence of meta data of a certain key with SlaveBase::hasMetaData().
Bernd Gehrmann [email protected]