The pfSense Store

Author Topic: Documenting Package System  (Read 48542 times)

0 Members and 1 Guest are viewing this topic.

Offline raj2569

  • Newbie
  • *
  • Posts: 22
  • Karma: +0/-0
    • View Profile
Documenting Package System
« on: April 05, 2006, 10:12:26 am »
Hi,

I am trying to write a package for clamav and havp. I have setup a test repository locally and trying to understand the manifest and package xml formats. Rather than just ask the questions, I will document what I understand and ask what I do not know, In the process making a crude documentation of package system. I will wikify it  once I get enough information.  Pl correct any errors

pfSense Packaging System

The basic idea behind pfSense packaging system is to install the normal FreeBSD packages, but keep the configuration inside the pfSense xml. This is achieved by creating the config file of application from scratch from xml duing installation and when it is changed in web interface. pfSense provides infrastructure to create the web interface and to store it in it's xml config. The package writer is expected to convert the data from xml to the native format of the application.

Creating a Local Repository.

To test local packages you need to create a local repository of packages. The local repository can be any web server accessible from wan interface with 2 main subdirectories. They are packages and pfSense. packages contain the main package manifest file pkg_config.xml. This also has config subdirectory for package specefic configuration files. The pfSense directory has xmlrpc code. Get the xmlrpc code from http://www.pfsense.com/xmlrpc.tgz and untar it in the directory. To change the default repository directory from pfsense.com to your webserver, edit the /etc/inc/globals.inc and change xmlrpcbaseurl to your repository. Now click the System -> Packages from the pfsense web interface and you should see packages from your local repository. You are set to go now.

Package System

pfSense package are composed of
  • Manifest File
  • Package Configuration files
  • Actual Binaries from FreeBSD

Manifest File

Manifest file is the packages/pkg_config.xml It contains list of packages available for installation and some basic information about packages. The format of manifest xml is as follows:

<pfsensepkgs>
<packages>
  <package>
  <name></name>
  <website></website>
  <descr></descr>
  <category></category>
  <config_file></config_file>
  <depends_on_package_base_url></depends_on_package_base_url>
  <depends_on_package></depends_on_package>
  <version></version>
  <status></status>
  <maintainer></maintainer>
  <configurationfile></configurationfile>
  <logging>
   <facilityname></facilityname>
   <logfilename></logfilename>
   </logging>
  </package>
 <packages>
<pfsensepkgs>


Package configuration files
Package manifest file for each package specifies a configuration file for each package via config_file tag. The convention is  to keep this file inside the packages/config/ directory. The format is:

<?xml version="1.0" encoding="utf-8" ?>
<packagegui>
   <name></name>
   <version></version>
   <title></title>
   <include_file></include_file>
   <!-- For what is this used? -->
   <backup_file></backup_file>
   <aftersaveredirect></aftersaveredirect>
   <!-- the package configs are stored in this subtree of main config file -->
   <configpath></configpath>

   <!-- pfSense menu hookup. Section should be an existing pfSense section -->
   <menu>
      <name></name>
      <section></section>
      <configfile></configfile>
      <tooltiptext></tooltiptext>
      <!-- the act=edit will let you directly edit the page -->
      <url>/pkg_edit.php?xml=package.xml&amp;act=edit&amp;id=0</url>
   </menu>

   <!-- Start the service, rc file comes from FreeBSD package or generated using write_rcfile -->
   <!--is rcfile and executable mutually exclusive? -->

   <service>
      <name></name>
      <rcfile></rcfile>
      <executable></executable>
   </service>

   <!-- For creation of Tabs inside the package page -->
   <!-- The page opened after clicking the menu item will have these tabs. -->
   <!-- url can be of the form  /pkg.php?xml=spamd.xml for getting data from a form -->
   <!-- Additional xml files should be included via additional_files_needed -->

   <tabs>
      <tab>
         <text></text>
         <url></url>
         <active/>
      </tab>
   </tabs>

   <!-- Used to include additional xml files for tabbed menu and (what all?)-->
   <additional_files_needed>
      <prefix></prefix>
      <chmod></chmod>
      <item></item>
   </additional_files_needed>

   <!-- What is this used for? -->
   <adddeleteeditpagefields>
      <columnitem>
         <fielddescr></fielddescr>
         <fieldname></fieldname>
      </columnitem>
   </adddeleteeditpagefields>

   <!-- data fields are defined here, these data is accepted from user-->
   <fields>
      <field>
         <fielddescr></fielddescr>
         <fieldname></fieldname>
         <description></description>
         <size></size>
         <type></type>
      </field>
   </fields>

   <!-- these gets executed during various times of package installation.-->
   <!-- Is the full list available here? -->


   <!-- This is used to define inline php functions that are called by other hooks -->
   <!-- You can also use include_file to include php code -->

   <custom_php_global_functions>      </custom_php_global_functions>

   <!-- The install and deinstall commands, create your directories etc here and call sync function -->
   <!-- Sync function should convert the xml to config file, write out the config fle and restart the daemon -->

   <custom_php_install_command>       </custom_php_install_command>
   <custom_php_deinstall_command>      </custom_php_deinstall_command>

   <!-- No idea where this is used -->
   <custom_add_php_command>      </custom_add_php_command>
   <custom_add_php_command_late>      </custom_add_php_command_late>
   <custom_delete_php_command>      </custom_delete_php_command>
   <custom_php_resync_config_command> </custom_php_resync_config_command>

   <!-- Is this similar to Service tab? -->
   <start_command> </start_command>

   <!-- When is this used ? -->
   <process_kill_command> </process_kill_command>
</packagegui>


Binaries from FreeBSD
The actual bineries are the normal FreeBSD packages for that particular program. That can be downloaded directly from FreeBSD FTP Server.

Misc

I will expand this as I proceed with my package and understand more about package system, In the mean time pl feel free to add more info as comments.

With Regards,

raj
« Last Edit: May 16, 2006, 02:49:42 am by raj2569 »

Offline raj2569

  • Newbie
  • *
  • Posts: 22
  • Karma: +0/-0
    • View Profile
Re: Documenting Package System
« Reply #1 on: April 06, 2006, 08:25:01 am »
Adding my questions here, if some one know the answers to them pl reply.

  • what is  the meaning of logging along with facilityname and logfilename in manifest xml
  • What is the difference between config_file and configurationfile in manifest xml
  • http://pfsense.com/packages/pkg_config.xml has package tag inside packages tab and ouside it. for example spamd comes inside packages tag,
    while pure-ftpd comes directly under pfsensepkgs. What's the difference?
  • Which of the tags are mandatory for manifest xml
  • Is the version tag in package xml the same as the one in manifest xml?
  • What types of file can be included via include_file ? (Only php or another xml file also?)
  • How is include_file different from additional_files_needed
  • When tabs are used what should be the format of externel xml files that are included?
  • How can I display the output of commands executed during installation to the output window?
  • How can I make one pfSense package dependent on another pfSense package? For example havp package depends on clamav package, and that package needs to be installed and configured
Code: [Select]
Apparently this is not possible[/list]
« Last Edit: May 18, 2006, 08:38:03 am by raj2569 »

Offline hoba

  • Hero Member
  • *****
  • Posts: 5837
  • Karma: +8/-0
  • What was the problem to this solution again?
    • View Profile
    • pfSense
Re: Documenting Package System
« Reply #2 on: April 06, 2006, 08:36:46 am »
Feel free to start a new section "Package Developement" at http://doc.pfsense.com  ;)

Offline raj2569

  • Newbie
  • *
  • Posts: 22
  • Karma: +0/-0
    • View Profile
Re: Documenting Package System
« Reply #3 on: April 06, 2006, 08:51:25 am »
Feel free to start a new section "Package Developement" at http://doc.pfsense.com  ;)

This will go there, if you notice I am updating the first post with new material as soon as I get them. But these needs to be peer reviewd to be posted as "Documentation" :) So post the answers ;)

raj

Offline lsf

  • Wireless Expert
  • Hero Member
  • *****
  • Posts: 3263
  • Karma: +5/-0
    • View Profile
Re: Documenting Package System
« Reply #4 on: April 17, 2006, 09:43:56 pm »
You might find this interesting if you didn't already see it.

http://wiki.pfsense.com/wikka.php?wakka=CoreGUI
-lsf

Offline billm

  • Hero Member
  • *****
  • Posts: 731
  • Karma: +1/-1
    • View Profile
    • UCSecurity - Technology discovery and ramblings
Re: Documenting Package System
« Reply #5 on: April 23, 2006, 09:05:11 am »
Adding my questions here, if some one know the answers to them pl reply.

  • http://pfsense.com/packages/pkg_config.xml has package tag inside packages tab and ouside it. for example spamd comes inside packages tag,
    while pure-ftpd comes directly under pfsensepkgs. What's the difference?

Disabled (non-working generally) packages are placed outside the packages tag.  Only question I know the answer to I'm afraid.  Wish I could get this running on localhost :-/  My box is being pissy with me ATM.

--Bill
pfSense core developer
blog - http://www.ucsecurity.com/
twitter - billmarquette

Offline billm

  • Hero Member
  • *****
  • Posts: 731
  • Karma: +1/-1
    • View Profile
    • UCSecurity - Technology discovery and ramblings
Re: Documenting Package System
« Reply #6 on: April 23, 2006, 09:58:04 am »
One note...if using a developers pfSense install to host the package files and the xmlrpc server, you will need to comment out line 394 in xmlrpc_server.inc.

It's the line that reads:
header($this->server_headers);

For some reason it seems to clash with lighttpd and I don't feel like debugging it.  Works fine with Apache apparently (and works fine w/out that header being sent).  With lighty, multiple headers appear to be sent.

--Bill
pfSense core developer
blog - http://www.ucsecurity.com/
twitter - billmarquette

Offline raj2569

  • Newbie
  • *
  • Posts: 22
  • Karma: +0/-0
    • View Profile
Re: Documenting Package System
« Reply #7 on: May 08, 2006, 07:16:15 am »
You might find this interesting if you didn't already see it.

http://wiki.pfsense.com/wikka.php?wakka=CoreGUI

Thanks for the link.

raj

Offline raj2569

  • Newbie
  • *
  • Posts: 22
  • Karma: +0/-0
    • View Profile
Re: Documenting Package System
« Reply #8 on: May 18, 2006, 07:39:45 am »
if you mess up thepkg_config.xml you will get a totally misleading error when you take System -> Packages. which says to check your network connection.

Code: [Select]
 
Unable to communicate to my_server.com. Please check DNS, default gateway, etc.

But this error can also come if your pkg_config.xml  is bad. I have spend the whole day today to track this :(

raj

Offline raj2569

  • Newbie
  • *
  • Posts: 22
  • Karma: +0/-0
    • View Profile
Re: Documenting Package System
« Reply #9 on: May 18, 2006, 07:41:35 am »

Offline sullrich

  • Hero Member
  • *****
  • Posts: 5110
  • Karma: +7/-2348
    • View Profile
    • pfSense

Offline raj2569

  • Newbie
  • *
  • Posts: 22
  • Karma: +0/-0
    • View Profile
Re: Documenting Package System
« Reply #11 on: May 19, 2006, 01:01:43 am »
http://your_server.com/pfSense/pkg_tester.php is your friend!

raj

Say what!?

I was using pkg_tester.php while I was debugging the the problem caused by incorrect pkg_config.xml. The problem exact problem was that I put two log statements in logging section. Like this:
Code: [Select]
          <logging>
                <facilityname>HAVP Access Log </facilityname>
                <logfilename>access.log</logfilename>
                <facilityname>HAVP Error Log </facilityname>
                <logfilename>havp.log</logfilename>
          </logging>

I was just trying out things by trial and error, and if I put two <facilityname> and <logfilename> inside <logging> the xml_parse($xml_parser, $data, feof($fp))) inside parse_xml_config_raw fails to return. To track this problem I used pkg_tester.php. that's why I said pkg_tester.php is your friend. If some one else is struggling with similar problems again this just gives them one more idea to work with.

raj 

Offline sullrich

  • Hero Member
  • *****
  • Posts: 5110
  • Karma: +7/-2348
    • View Profile
    • pfSense
Re: Documenting Package System
« Reply #12 on: July 15, 2006, 01:25:21 pm »
For my records in the future:

Warning:  Header may not contain more than a single header, new line detected. in /usr/local/www/pfsense/Etomite0.6/pfSense/xmlrpc_server.inc on line 394

This fixes this issue.  Just ran into this on the pfSense.com website serving packages.

Woops.

Offline Thowie

  • Newbie
  • *
  • Posts: 9
  • Karma: +0/-0
    • View Profile
    • Networkmonkey IT Blog
Re: Documenting Package System
« Reply #13 on: September 26, 2006, 01:01:57 pm »
Hello,
I made a local "Package server" on my Network for Develop on packages. I do all this what you write in your config, and it works. But if in the pkg_config.xml a package has more than one <depends_on> lines, like squid or clamav, i became an errormessage on my pfsensebox. He says: "cant connect to xxx.xxx.xxx.xxx. Cached data yoused." If i only use packages that had only one <depends_on> line, all works fine. Why is that so???
The Packageserver is a Apache 2 Server with the latest xmlrpc-code.
Do i somthing wrong???
So please help me, because my dsl-connection somtimes break down for hours. Not good :(
So thanks and a nice day for you...
Bye
Thomas

p.s. sorry, for my worst english ;)

Offline captaintrip

  • Newbie
  • *
  • Posts: 1
  • Karma: +0/-0
    • View Profile
Re: Documenting Package System
« Reply #14 on: November 11, 2006, 10:35:28 am »
hi thowie,

you have to checkout the xmlrpc stuff from the cvs (www folder) - the xmlrpc.tgz in the first post is an older version.
then it works with more than one depency per package.

greets
captaintrip

Offline ermal

  • Hero Member
  • *****
  • Posts: 3832
  • Karma: +85/-5
    • View Profile
Re: Documenting Package System
« Reply #15 on: July 09, 2009, 09:30:10 am »
Adding my questions here, if some one know the answers to them pl reply.

  • what is  the meaning of logging along with facilityname and logfilename in manifest xml
  • What is the difference between config_file and configurationfile in manifest xml
I cannot tell you a clear answer to this i have to look at the code.

Quote
  • http://pfsense.com/packages/pkg_config.xml has package tag inside packages tab and ouside it. for example spamd comes inside packages tag,
    while pure-ftpd comes directly under pfsensepkgs. What's the difference?
Some packages are removed from that list to not be allowed to install iirc.

Quote
  • Which of the tags are mandatory for manifest xml
Only logging/descr is optional, while missing the others might break things.

Quote
  • Is the version tag in package xml the same as the one in manifest xml?
Yes.

Quote
  • What types of file can be included via include_file ? (Only php or another xml file also?)
include_file is just for including a file that is needed and it need to be php files.

Quote
  • How is include_file different from additional_files_needed
additional_files is used during installation/syncing of package while include is for needed dependencies of php code.

Quote
  • When tabs are used what should be the format of externel xml files that are included?
XML the same as the xml file you use for one package.

Quote
  • How can I display the output of commands executed during installation to the output window?
There is a global variable $static_output to which you can add.

Quote
  • How can I make one pfSense package dependent on another pfSense package? For example havp package depends on clamav package, and that package needs to be installed and configured
Code: [Select]
Apparently this is not possible
    Its through additional_files_needed tag but i think it is not really finished or has some culprits to use.
    Basically you specify the xml files in that tag that are needed.