Lessons in Packaging Linux Applications
On packaging
For Free Software, more users means more potential developers. With tvtime, we made an effort to ensure high quality packages for all major Linux distributions. This article describes some of our findings when packaging for ALT Linux, Debian GNU/Linux, Gentoo Linux, Mandrake Linux, Red Hat Linux, Slackware Linux, and SuSE Linux.
We present some of the lessons we learned and some of our assumptions that we found to be false. We believe our project is typical in that it is fairly young and unknown, and so we believe that many other developers and packagers may benefit from these notes.
Packager-developer communication is poor
We originally thought that finding packagers for our application would be difficult. However, we soon discovered that packages were being made, but we were rarely notified. While some of these packagers had just built RPMs for their own personal use, we found that many had made packages which they provided to the community or contributed to their distributions. Our Mandrake packager explained that he never contacted us because he never had any problems with our package.
Our advice to application developers:
- Routinely use Google to search for your own application. You will likely find packages made by users.
- Check the contents of packages. We mostly use Debian, but have become quite familiar with tools such as rpm2cpio, as well as the .spec and .ebuild file formats. Make sure that no files are missing.
- The "packager" tag in RPM files has different semantics depending on the distribution. Be sure to look in the spec file and see who the maintainer is, not just the person who touched it last. The email addresses listed in the changelog should be trusted over any other tags.
- If you find a package that looks official, try looking for it on the distribution QA site. This may give more detailed contact and bug reporting information. Both Debian and Mandrake have QA sites.
For packagers:
- We strongly recommend that you include your email address somewhere in the package. This is especially awkward in the case of Slackware packages or Gentoo ebuilds which are often missing this information. Developers may wish to contact you to help fix errors in your packaging.
The lack of email messages or other notifications from packagers was by far our most surprising observation, and we believe that this issue deserves more awareness. While many packagers have assured us that they usually do send an email to the developers, we hope that developers will make package searching a part of their own QA process.
References:
Errors in packaging
We have learned to watch for some specific errors in the packages during a new release of our software. We believe that these issues are usually noticed more easily by the developer than the packager. Developers should:
- Check for missing documentation. GNU autoconf provides no standard location for HTML documentation or example files, so packagers must install this separately. New documentation files can easily be missed.
- Check the package description and URLs. If these have been updated to mention new features, ensure that the package information matches.
- Watch for out-of-date dependency information. Packagers tended to find new dependencies, but forgot to check for removed ones.
Dependency problems
A problem for packaging any program is how to handle non-standard dependencies. There were three main problems we encountered:
- If you depend on library packages not in the standard installation of a major distribution, distribute the packages for those from your Web site.
- If the dependency is optional, some packages may be built without it. Make sure that both you and your users know about crippled packages.
- Watch for incompatible library packages. For us, it was lirc, the Linux Infrared Remote Control library. It seems there are incompatible packages of the lirc client library being distributed for Red Hat.
Clearly, there is never an ideal solution for these sorts of problems, and it's hard to give advice about what to do with them. We do not believe that the responsibility for these sorts of difficulties can lie completely with the packager. We believe that applications themselves can often be improved by having runtime (vs. compiletime) options and even by avoiding dependencies that are problematic for users. However, in order to make these judgements appropriately, feedback is required from packagers on multiple distributions.
Menu entries and .desktop files
The most common thing that packagers of tvtime added to our packages was a menu entry or .desktop file. We discovered that by consolidating the combined input from our packagers, we could help ensure that our menu entries had a similar look and feel across distributions and menu systems.
The .desktop file format
The desktop entry file format standard is used by both GNOME and KDE for their menu entries. However, installation locations and conventions for both the .desktop files and associated icon files have not been standardized until recently. freedesktop.org is leading this effort with standards for menu entries and how applications can locate them. This standard is used in KDE 3.2 and GNOME 2, but the rollout of these systems is not yet complete.
We found that understanding what standards exist and how we should use them was quite challenging. Where should the .desktop files be placed by a .spec file or Debian installation script, do we need to install them separately for a GNOME or KDE setup, and how is this affected by user upgrades? We lacked any Best Current Practices information from desktop environments or distributions.
Here is a list of where we decided to place our .desktop files and icons for installation on modern distributions. We don't believe this list is sufficient and would like to expand it to include more common cases.
- $datadir/applications/ for our .desktop file and $datadir/icons/hicolor/size/apps/ for our icons. This follows the freedesktop standard.
- $datadir/pixmaps for our icons. This fixes icons on older versions of GNOME and on Mandrake.
- On SuSE, we link to our .desktop file from /opt/kde3/share/applnk/ and include our icons in /opt/kde3/share/icons/.
- On Debian, we link to our .desktop file from /usr/share/applnk/.
References:
- http://www.freedesktop.org/standards/desktop-entry-spec/
- http://www.freedesktop.org/standards/menu-spec/
The Debian menu system
Debian created its own menu format for applications which predates the freedesktop work. This menu format is also used by Mandrake, but it is slightly modified. ALT Linux uses the same modified system from Mandrake.
The format is simple, but less powerful than .desktop files. The most notable disadvantage of this format is the lack of internationalized names. Our entry for Debian is:
?package(tvtime):needs=x11 section=Apps/Viewers \
title="tvtime" command="/usr/bin/tvtime" \
icon="/usr/share/pixmaps/tvtime.xpm"
For both Mandrake and ALT Linux, we use:
?package(tvtime):needs="X11" section="Multimedia/Video" \
title="tvtime" command="/usr/bin/tvtime" icon="tvtime.png" \
longtitle="High quality television application"
The section titles are completely different between Debian and Mandrake. For our application, a TV viewer, the Mandrake sections are much more descriptive. Also, it seems that Debian requires icons in the xpm format.
Since the freedesktop initiative is so promising, these legacy menu systems may soon disappear, so we recommend that maintaining these menu entries be done by the Mandrake and Debian packagers. However, we strongly recommend that the upstream ensure that both the Mandrake and Debian menu entries are as close to each other as possible.
References for these files from Mandrake and Debian are here:
- http://www.linux-mandrake.com/en/howtos/mdk-rpm/advanced.html
- http://www.linux-mandrake.com/en/howtos/mdk-rpm/mdk-menu-structure.html
- http://www.debian.org/doc/packaging-manuals/menu.html/
- http://www.debian.org/doc/packaging-manuals/menu-policy/
Hearing about bugs and dealing with them
Probably the most difficult problem for users and developers alike is the handling of bug reports. This includes both bugs in the application and in the packaging. We found only Debian to provide us a method of automatically receiving all bug reports for our application (through http://packages.qa.debian.org/ and the package tracking system). However, Red Hat does allow you to subscribe by maintainer.
Many believe that the distribution QA infrastructure should provide the first level of tech support in all cases, and users should always be instructed to seek support first from their distributions. We believe this is irresponsible. Most packagers and distribution QA teams, like developers, are volunteers with time constraints, and application developers need to be aware of packagers and their packages since they will likely have to help support them.
We would like to see packagers inform developers about the user experience of their packages. At minimum, we would like to know:
- How do users install your package in their distribution?
- Where is the documentation installed? Debian uses /usr/share/doc/tvtime/, Mandrake uses /usr/share/doc/tvtime-0.9.10/, and Official Red Hat packages use /usr/share/doc/tvtime-0.9.10-5/, replacing 5 with the release number of the package.
- Is there someplace users should go to report bugs in your package?
- How can we subscribe to be notified of bugs?
This information can go a long way towards helping developers provide better support for their applications.
References:
- http://packages.qa.debian.org/
- http://www.debian.org/doc/developers-reference/ch-resources.en.html#s-pkg-tracking-system
- https://bugzilla.redhat.com/bugzilla/userprefs.cgi?tab=email
Conclusion
We have presented some of our experiences in packaging our application and consolidating our user experience across many Linux distributions. We hope this will help packagers, distributors, and other developers be more aware of the problems and possible solutions for distributing Linux software effectively.
We also believe that packagers can do more to tell developers what might make their jobs easier. We found that through a small amount of coordination, we could consolidate some common additions done by different packages, reducing work for the packagers. We believe that this significantly reduces the number of distribution-specific bugs.
Acknowledgements
Our thanks go to Oliver Baltzer, Mike Harris, Sam Hocevar, Max Kalika, Simon Law, David Lehn, Guillaume Rousse, and Matt Zimmerman for their comments on this document.
Lessons in Packaging Linux Applications
Very good, but I noticed you didn't even touch on the issue of proper file and directory access permissions and security considerations. I think this is obviously important and needs to be considered by any and all package builders and developers also.
As a side note, one of the main reasons my favorite distro (Slackware) is my favorite distro is because building packages seems so much simpler and easier than other distributions I've used. I hope that package creation can be kept reasonably simple. There is nearly always a tendency to add feature-bloat and more and more complication to standards as well as software.
In the case of packages for different distributions it is my opinion that the guidelines should be kept as simple as possible, if not simpler.
Hey, I admit it. I'm simple.
Creating packages should not become some arcane art that only a 'priesthood of packagers' can accomplish properly. It needs to be something anyone can do, even if their package is only for their own particular system.
Cheers -
Chuck Bell
mandrake menu system
check your facts, the mandrake menu system is internationalized.
Re: mandrake menu system
> check your facts, the mandrake menu
> system is internationalized.
>
I did some research and discovered that there is an mdk-menu-messages package which contains internationalized text for many menu system strings. I did not know about this, thanks.
However, that doesn't really help the problem. This packages is specifically for Mandrake's own strings. This won't let you give many different strings for your application name or menu entry text in different languages without also going through that package. I don't believe this is practical for all applications to provide i18n'ified menu entries. I would like to hear your opinion on it.
The .desktop format is more flexible in that you can provide translations directly in the file provided by our package.
Re: Lessons in Packaging Linux Applications
> Very good, but I noticed you didn't even
> touch on the issue of proper file and
> directory access permissions and
> security considerations. I think this is
> obviously important and needs to be
> considered by any and all package
> builders and developers also.
You are right, it would have been useful to touch on this subject. I have not seen many permissions problems across distributions, except for the directory in /var/run we create to place our FIFOs, much like screen would use (although it is an interesting exercise to see where screen's FIFOs are placed in different distributions). This directory should be group writable by any users that run tvtime, usually anyone that can access the /dev/video devices. Getting these permissions correct was another part of the packaging where we have found errors.
The obvious solution.....
... is to have developers build their own packages, which are the official way to install the software. The team that wrote the software are the best qualified to package it anyway.
PIMP ALERT: so if you have some spare time, dear reader, come help us build autopackage, the intention of which is exactly that :)
Fedora
www.fedora.us have a really good packager community , atleast for RedHat. Check it out.
Mandrake wiki
The mandrake rpm howto is now hosted on the
mandrake wiki.
http://qa.mandrakesoft.com/twiki/bin/view/Main/MandrakeMenu
Packaging, an alternative perspective
The concept of packaging, in an environment which is highly fluid, may actually be a mistake.
I am going to propose an alternative view, which might address some of the problems currently facing packaging systems.
Instead of specifying a directory tree, specify file groupings. Have the destination of those groupings be decided by the installer software or installee. That way, you don't have the issue of incompatiable installation trees - the tree will always be (by definition) compatable with the user's preferred structure.
Instead of specifying file permissions, specify file natures. MIME types should be adequate for the job. After all, you don't know what permission system is being used. Standard UNIX? POSIX ACLs? Trustees ACLs? MACs? Since the "package" builder doesn't know, they can't tell the package exactly what setup to use. All they can do is describe what they want. The installer software needs to be able to parse that description into something usable by the permission system that is installed.
Dependencies are a huge problem with packages, partly because absolute version numbers are used, or minimum versions, or no versions at all. Nor is any information included on how to meet those dependencies adequately. (eg: You might need PHP 4.1, but do you say what modules are required?) It's no use filling the dependencies on first match, either, because of the configuration issues. You need to parse ALL dependencies, decide what the COMPOSITE requirements are, and then (and only then) actually try to resolve those requirements. However, you've still got the problem that those requirements may be dependent on each other, too. You must therefore gather all of the required components, determine THEIR composite requirements, and repeat, until a complete picture has been assembled. This must also be matched against requirements of existing software, so that you don't break a pre-existing dependency. Only after all these stages are you ready to install/build the necessary components, in the necessary order.
Incompatiable requirements can be resolved, by various means. One method is to use group permissions. Create assemblies of packages, and then use group permissions to split the complete collection of packages into compatiable sub-collections (ie: the assemblies). eg: Program A is required by program B, but conflicts with program C which is required by program D. Put A and B in the same group, C and D in the same group (but different from A and B), then set A and C to be group executable but not world executable. (Or, if you're using ACLs, simply ensure A and C are never on the same list.)
Although you can resolve certain incompatibilities using the above method, it has its limits. The package software and package maintainer must therefore place certain constraints on conflict resolution, and define actions if those constraints would be violated. eg: Is the package to be installed, or is the conflicting package to be removed?
Once you've built up a system for handling revision control, conflict resolution and dependency resolution, you've got a packaging system that's actually worth using.
Until then, it will always be possible to package things up correctly, and yet still produce badly broken results. In today's day and age, that should not be acceptable. We've more than enough resources to handle all possible contingencies in a controlled, robust manner.
But, hey, there's always the case that that wouldn't be nearly as fun as the chaos.
One thing I'd love to add to RPM.......
And it would have a tremendous affect on the ease of building cross,rpm based, distiribution packages. A logical or. Right now if say RedHat calls it libFoo, Mandrake has libfoo and libfoo-devel and SuSE just has libfoo then you can't build a single rpm. but if you could say.
depends = libFoo || libfoo > 1.4
then one rpm would install on all 3 systems. The MDK/Debian unified menu structure (same menu across all dm's) is a boon but the real problem is the need for a logical or. This could also be used in a case where any one of 2 or more rpms supply the same file. (say a .h file)
depends = libfoo > 1.4 || libbar > 2.0 || ralph > 2.1
We have the ability to do a logical and, greater than, less than and exclusion but why the or is missing I don't know. (and yes I've written to the rpm.org people.)
fogoten Distribution
Another well-known distribution that was not
mentioned in the article is
ROCK Linux a disrtibution build kit with
predefined distributions.
We automatically generate dependencies - and
defined a rock-registry where custom .desktop
files are placed to generate common menues for
any window-manager / desktop environemtn you
need.
Re: Lessons in Packaging Linux Applications
> As a side note, one of the main reasons
> my favorite distro (Slackware) is my
> favorite distro is because building
> packages seems so much simpler and
> easier than other distributions I've
> used. I hope that package creation can
> be kept reasonably simple.
> [...]
> Creating packages should not become some
> arcane art that only a 'priesthood of
> packagers' can accomplish properly. It
> needs to be something anyone can do,
> even if their package is only for their
> own particular system.
Have you seen http://checkinstall.izto.org? I use it to create my own debian
packages: ./configure; make; make install
and you have a .deb package. The nice thing is you
can add dependencies to them, and last time I
upgraded with apt-get upgrade, my custom
made packages were replaced correctly with new
binary versions that were available.
Really fantastic. Do get the development version
for dependencies and other advanced uses which
still have not made it to the stable branch.
Re: The obvious solution.....
> ... is to have developers build their
> own packages, which are the official way
> to install the software. The team that
> wrote the software are the best
> qualified to package it anyway.
I can't agree with you. All distributions have their little differences (well, some of them are little, some are not), and software developer must mention all this differences. For example, packaging policy (menu groups, documentation placement, package nameing, in case of RPM - various distribution-specific specfile macros), security issues (PAM modules, control(8) like in Owl and ALT). And developer still must do further development...
Developer is qualified to develop software and packager is qualified to package this software for his distribution.
Re: One thing I'd love to add to RPM.......
> depends = libFoo || libfoo > 1.4
Well in some cases the same effect can be achieved by means of %if but it's definitely less clean.
I find myself using it when doing packages which have to build on different generations of the distro which can differ in some aspects.
Re: Lessons in Packaging Linux Applications
> (although it is an interesting exercise
> to see where screen's FIFOs are placed
> in different distributions).
Our screen fifos live in ~/tmp/screen/ :)
trickster:~> echo $TMPDIR
/home/mike/tmp
> This directory should be group writable by
> any users that run tvtime, usually
> anyone that can access the /dev/video
> devices.
On systems using pam_console(8), it's commonly not group-based, at least without administrator intervention.
I was thinking on it while deciding how to restrict access to SUID binary -- for now it's under control(8) (object state persistency system employed in Owl and ALT) but the most proper way would be to ensure it's local console user which will be given /dev/video access (and audio out). Seems like it's consolehelper way, didn't look into that for a long time though.
Re: The obvious solution.....
> ... is to have developers build their
> own packages, which are the official way
> to install the software. The team that
> wrote the software are the best
> qualified to package it anyway.
Nope. Should I elaborate on *why*?
Re: Packaging, an alternative perspective
> The concept of packaging, in an
> environment which is highly fluid, may
> actually be a mistake.
[skip]
You seem to overestimate the actual problems and underestimate what is already in place regarding dependencies whatsoever (to me).
E.g. we use automated tool to find what's being probed for and used during package build, and ensure the repeatability of the build since it's being done in isolated chroot with only announced dependencies recursively installed (resolved by apt-rpm).
It's not perfect, but good enough to provide real-world users with software in good shape, and real-world maintainers -- with less pain and frustration than usually.
And for me, packages plus build-time insulation is the only _practical_ way to ensure given properties of the result and its repeatability: this way we fixate the properties of all the code which has to be in place for our one to build, and can concentrate on fiddling with it and not its environment when we don't need that. (I doubt it's practical to do source-build of build environment itself -- even installing pre-built packages is not optimal performance-wise but still quite bearable)
Ummm... and still badly broken results can be achieved anyway! :-)
Re: One thing I'd love to add to RPM.......
>
> % depends = libFoo || libfoo > 1.4
>
>
>
> Well in some cases the same effect can
> be achieved by means of %if but it's
> definitely less clean.
>
> I find myself using it when doing
> packages which have to build on
> different generations of the distro
> which can differ in some aspects.
>
>
Hmmmmm. this is an intresting work around I'll have to try it. Thanks.
Re: Lessons in Packaging Linux Applications
> Have you seen
> http://checkinstall.izto.org?
I've only a couple of "problems" whith this packaging tool as far as making rpms goes.
1. It doesn't allow of the inclusion of "extra" files such as say an init.d script.
2. It doesn't allow for the creation of a src rpm.
3. It doesn't include changelog editing.
However for a self created rpm so that I can (very) easily remove/upgrade a non packaged program, I've found it to be a neccesary tool in my toolkit. For the creation of locally distributed rpms, it's easy, reliable, and so far, it hasn't broken anything.
DeMorgan to the rescue?
> We have the ability to do a logical and,
> greater than, less than and exclusion
> but why the or is missing I don't know.
> (and yes I've written to the rpm.org
> people.)
First, I assume you mean logical NOT when you say exclusion.
If you can do AND and NOT, why can't you simulate OR?
A OR B = NOT ( ( NOT A ) AND ( NOT B ) )
It certainly doesn't look as clean, but it is logically equivalent.
Of course, I don't know anything about RPM .spec files. Does exclusion mean NOT? Is it possible to string together NOTs and ANDs? If yes to both, then you should be able to recreate the above logic.
Re: DeMorgan to the rescue?
>
> % We have the ability to do a logical
> and,
> % greater than, less than and exclusion
> % but why the or is missing I don't
> know.
> % (and yes I've written to the rpm.org
> % people.)
>
>
> First, I assume you mean logical NOT
> when you say exclusion.
>
> If you can do AND and NOT, why can't you
> simulate OR?
>
> A OR B = NOT ( ( NOT A ) AND ( NOT B )
> )
>
> It certainly doesn't look as clean, but
> it is logically equivalent.
>
> Of course, I don't know anything about
> RPM .spec files. Does exclusion mean
> NOT? Is it possible to string together
> NOTs and ANDs? If yes to both, then you
> should be able to recreate the above
> logic.
Actually I really did mean exclusion. Or perhaps a better choice would have been conflict creation. In a spec file I can say that if X is installed Y cannot be installed at the same time. For example if proftpd is installed wu-ftpd could not be installed without first removing proftpd. The logical not is really an implied not in that by using the exclusion characteristic or the > and
Dependancy Issues
A lot of dependancy issues will magically go away when you compile your binaries as static.
Sure, your binary file will become large, but not significant enough for a user to decide not to use your package.
Have any of you had experiences with this kind of thing?
Re: Lessons in Packaging Linux Applications
as you know...my favorite distro is redhat. may be it is sound subjective because i've used it since 1998 and it's very user friedly. Package is very interesting thing, because for me, choosing an approriate OS, sometime can be see when we do the installing. I think, if the distro make the package simpler and easier, alotof people will choose it. And that what microsoft do the best. :P. if linux can make it, wow, the great OS with Great installer, it will make linux really unbreakable.
the only barriers linux and the user is, linux is very difficult. that's we have to end it now.
>
> % As a side note, one of the main
> reasons
> % my favorite distro (Slackware) is my
> % favorite distro is because building
> % packages seems so much simpler and
> % easier than other distributions I've
> % used. I hope that package creation
> can
> % be kept reasonably simple.
> % [...]
> % Creating packages should not become
> some
> % arcane art that only a 'priesthood of
> % packagers' can accomplish properly.
> It
> % needs to be something anyone can do,
> % even if their package is only for
> their
> % own particular system.
>
> Have you seen
> http://checkinstall.izto.org? I use it
> to create my own debian
> packages: ./configure; make; make
> install
> and you have a .deb package. The nice
> thing is you
> can add dependencies to them, and last
> time I
> upgraded with apt-get upgrade, my
> custom
> made packages were replaced correctly
> with new
> binary versions that were available.
>
> Really fantastic. Do get the development
> version
> for dependencies and other advanced uses
> which
> still have not made it to the stable
> branch.
Re: Dependancy Issues
> A lot of dependancy issues will
> magically go away when you compile your
> binaries as static.
>
> Sure, your binary file will become
> large, but not significant enough for a
> user to decide not to use your package.
>
> Have any of you had experiences with
> this kind of thing?
Yes. For one thing, if a security hole or bug is found in a library used by the application, you must recompile the library AND the application to fix it. It also presents complications in having different applications compiled with different versions of libraries (Application A depends on Application B, but Application B was compiled with earlier versions of Libraries A & B, so it does not support the operation Application A needs, so you must upgrade Libraries A & B and Application B, ...). Not to mention those bigger binaries take up more RAM unlike the dynamically-linked libraries which are only loaded into RAM once for multiple apps.
Statically linking your apps basically works against all the reasons they thought up dynamic linking in the first place. Don't do it.