[overview] | [download] | [library] | [shell overview] | [shell manual] | [utilities] | [faq] |
Modglue is a combination of several software tools that assist in writing and running programs connected to each other using standard Unix pipes. There are two main parts to it, the modglue library and the modglue shell. In more detail:Each of these is explained in more detail below. There are also several small additional programs bundled with modglue, such as a program to add readline capability to any command line program.
- modglue library
- This is a C++ library with classes for forking external processes and asynchronous reading from streams. It takes away the burden of all subtleties involving the Unix fork call. The asynchronous read facility enables one to read on multiple input streams at the same time, without loosing any of the standard C++ stream facilities.
- modglue shell
- A shell with special support for arbitrarily connecting process pipes together, in a way that goes beyond the standard Unix linear pipe idea (a bit like ksh co-processes, but much more powerful). In contrast to existing shells, you can start any number of processes and connect their stdin/stderr/stdout pipes in any way you like, not just linear as usual.
Moreover, the modglue library extends the idea of standard Unix pipes by providing a general framework for the creation of new processes with more input or output pipes. The modglue library has support for this built in, but the setup is general and can be used from any language.
In general, the idea behind modglue is that the Unix pipe concept is very powerful, but insufficient to construct interactive programs that require bidirectional communication between processes. The library and shell extend this concept in a completely backwards compatible way such that large robust systems can be built from individual Unix building blocks.
The idea of "circularly connected pipes" is not new, but usually only implemented in an ad-hoc fashion. See for instance satshell, which is a gui tool which reads gui building commands from stdin and writes event information (pressed buttons and so forth) to stdout. In this case, the satshell program itself is given a program to which it should connect its stdin and stdout. For another similar idea (though without implementation), see this Freshmeat editorial by David Symonds. Modglue separates the connection logic from the actual program (in the context of satshell, the gui tool would be one process and the script another, and modglue would connect their stdin/stdout pipes in a circular way).
Once Unix building blocks are more easily connected in order to build complex programs, one can perhaps start building graphical tools to do this (an example of this approach, though with custom building blocks instead of programs that stick to the Unix pipe philosophy, can be found in triana).
Then, download and install modglue.tar.gz. Follow the
instructions in the INSTALL file for installation details.
Some shells have ways to circumvent this limitation. The
korn
shell has so-called "co-processes", which you can start as
Moreover, there is no standard way
to extend the number of input/output pipes that a program can have.
(co-processes do not allow for input redirection to startup arguments; since many
programs can only take their arguments from argv and not from stdin, this is
(unfortunately) a requirement. Also, these guys do not provide a pseudo-tty so most
programs started as coprocesses will start buffering).
The modglue library takes the Unix concept one step further
and allows programs to open more than these three standard
pipes. Furthermore, the links between programs can be more flexible
(you can have complicated graph-like structures instead of just a
single linear pipe) and writing programs that listen on multiple
pipes in a non-blocking way is considerably easier. For instance,
a program could be prompting the user with a dialog for entering
a url, then send that request out through its output pipe,
and read in any responses through its input pipe.
The modglue library implements a strict superset of the standard Unix
pipe idea: all normal Unix programs can be used as modglue building
blocks, the only limitation being that these programs of course
have only the three standard pipes.
There are two types of programs in the modglue world: unix executables and
modglue executables. The second category is a strict superset of the first one; all
modglue executables are also unix executables, though they may not be able to offer full
functionality when run from a unix shell instead of the modglue shell.
If you try to start Unix executables this way, they will automatically
be wrapped using the "ptywrap" command.
Options can be given by adding them in a comma separated list
after the '^' character.
At the moment, the available ones are start_on_input
and abort_on_failed_write.
There is a shortcut possible if you just want to connect in the
normal Unix way (connect stdout and/or stderr of one program to stdin of the
next): just type the program names separated by a pipe symbol '|'.
Some of the functionality of modglue can be found in TCL, in particular the ability to
send data to the stdin of an already forked program. Modglue aims for
a much more basic goal: it just links programs together, nothing more.
In this sense it is a lot simpler than TCL (and it feels more like
a normal Unix shell).
The `other' component of the socket channel is used solely for status information,
and never directly visible to the user of the program. These status messages
are suppressed when modglue binaries run in Unix mode, so they do not interfere
with normal expected Unix pipe behaviour.
In case your program is going to write explicit requests for data
on a pipe and then listen for the reply on another one, it is good
to make the names related. Suggested are reqout and reqret
and variations thereof if you need more than one pair.
See the next question for info about the stream identifier field.
Obviously this header yields some overhead, and makes it clear that you
should try to avoid sending only a few bytes of data. Compared to something
like XML-RPC, the cost is still pretty minimal though.
Making an asynchronous request then goes as follows. Create a message with
a new identifier, and store the identifier somewhere. When input comes
in, compare its identifier to what was stored. If they match, it is a reply.
If you never block but instead use the main event loop to wait for
incoming data, you avoid all potential deadlock situations. Just
think differently.
In modglue, the user who starts processes decides which pipes it is
going to listen to. This puts a little bit more burden on the user,
but removes altogether a complicated layer of message distribution
algorithms. Modglue is thus simpler, at the expense of needing a
user willing to do a little bit more work.
Download and install
You need the following additional software in order to compile
and use modglue:
Modglue library
#include <modglue/modglue.hh>
modglue::pipe foopipe("foo", modglue::pipe::input, 0);
modglue::pipe barpipe("bar", modglue::pipe::output, 1);
void print(const string& txt)
{
cout << "received " << txt << " on foo pipe" << endl;
cout << "sending something on bar pipe" << endl;
while(txt!="end") {
string str;
if(!foopipe.read(str,2))
break;
}
barpipe.sender("thank you!");
}
int main(int argc, char **argv)
{
modglue::main mm(&argc, &argv);
mm.add(pipe1);
mm.add(pipe2);
foopipe.receiver.connect(slot(print));
mm.run();
}
Modglue shell overview
On Unix systems, the standard way to combine small
programs into bigger ones is to tie them together using pipes. As all
standard conforming programs have three standard pipes (stdin, stdout,
stderr) open, it is easy to feed the output of one program into
another one:
However, this way of connecting programs is limited in the sense that
data only flows one way (from the first program to the second and so
on, but never back).
wget -O - http://somewhere | grep "something"
after which you can read from and write to the standard pipes. A
similar concept exists in zsh,
where co-processes are started by pre-fixing them with the command
"coproc". Although very useful, these shells only allow for a single
co-process.
program |&
Modglue shell user manual
Modshell is the analogue of a normal unix shell like bash or csh. It
contains the logic to tie sockets of various modglue executables
together and spawn new processes. However, it is intended to be usable
as a replacement for a normal Unix shell, in the sense that running
programs in the normal Unix way is supported (with the standard
notation, eg. using the '&' character to start programs in the
background and the '|' character to connect stdout/stdin of
programs). In more detail, the options are:
> jobs
13446 running xbiff
xbiff::stdin
xbiff::stdout
xbiff::stderr
13447 running gtkiowin
gtkiowin::stdin
gktiowin::stdexc
13558 standby, start_on_input wget -O -
wget::stdin
wget::stdout
wget::stderr
Utilities
Several small programs are bundled with modglue. They provide some
additional functionality that was easy to implement with the library,
or are generally useful for command-line driven programs.
is all you need to know.
> ptywrap [unix executable]
Frequently asked questions
The reader will then be able to figure out whether to wait for more messages
to come which really belong to the same outgoing message.
field size (bytes version number and flags 1 stream identifier 8 length 4 data length
$Id: oldindex.html,v 1.1 2007/03/05 22:39:00 peekas Exp $