pfSense Gold Subscription

Author Topic: Working with Packages on 2.3 and Converting to Bootstrap  (Read 17205 times)

0 Members and 1 Guest are viewing this topic.

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21388
  • Karma: +1432/-26
    • View Profile
Working with Packages on 2.3 and Converting to Bootstrap
« on: December 03, 2015, 11:25:21 am »
I have collected and combined a lot of information and added some more to come up with what I hope is a good set of info to help others make the jump into working with packages on 2.3 and converting things to bootstrap as needed.

Basic info for those who have not yet done any development work:
https://doc.pfsense.org/index.php/Getting_Started_with_pfSense_Development
https://doc.pfsense.org/index.php/Contributor_License_Agreement_for_Developers

Overview of the new pfSense Package port directory structure:
https://doc.pfsense.org/index.php/Package_Port_Directory_Structure

A list of current packages on 2.3 and where they are located inside the ports repository:
https://doc.pfsense.org/index.php/Package_Port_List

Updated doc about Developing Packages with 2.3-specific info:
https://doc.pfsense.org/index.php/Developing_Packages

Notes about converting packages to bootstrap:
https://doc.pfsense.org/index.php/Converting_Packages_to_Bootstrap

2.3 Package Conversion and Operation Status
Here is a Google Docs sheet with the current list of 2.3 packages and their status. Many use XML forms and do not need any help (which is good!) but still need testing to confirm they function as expected on 2.3.

The master ticket in Redmine for the conversion of packages is here: https://redmine.pfsense.org/issues/5568

Let me know if anything seems to be missing or if there are any specific topics missed that need covered.

Thanks to all for their effort!
« Last Edit: December 03, 2015, 12:52:39 pm by jimp »
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline jdillard

  • Administrator
  • Sr. Member
  • *****
  • Posts: 534
  • Karma: +124/-1
  • Web Dude
    • View Profile
    • Personal Website
Re: Working with Packages on 2.3 and Converting to Bootstrap
« Reply #1 on: December 03, 2015, 11:55:28 am »
bmeeks, I was thinking of taking a shot at converting Suricata so I had somewhere to dive in if you need a hand.

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21388
  • Karma: +1432/-26
    • View Profile
Re: Working with Packages on 2.3 and Converting to Bootstrap
« Reply #2 on: December 03, 2015, 11:59:49 am »
I added a link above to a Google docs sheet with the current status of the packages we have moved to 2.3 already. Many do not need conversion but still could use testing.

Feel free to report success or failure here in this thread but if you do find a problem please start a new thread to investigate the problem to avoid crosstalk this thread.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3159
  • Karma: +818/-0
    • View Profile
Re: Working with Packages on 2.3 and Converting to Bootstrap
« Reply #3 on: December 03, 2015, 12:33:15 pm »
bmeeks, I was thinking of taking a shot at converting Suricata so I had somewhere to dive in if you need a hand.

Thank you!  I will be glad for the help and to help you.  Suricata has not been started. I had been working on Snort until I took a break to work on the netmap stuff.  Send me a PM and we can swap e-mail addresses (if you don't have mine already).  Will be easier probably to correspond that way or by phone even.

Bill

Offline jdillard

  • Administrator
  • Sr. Member
  • *****
  • Posts: 534
  • Karma: +124/-1
  • Web Dude
    • View Profile
    • Personal Website
Re: Working with Packages on 2.3 and Converting to Bootstrap
« Reply #4 on: December 03, 2015, 12:35:56 pm »
bmeeks, I was thinking of taking a shot at converting Suricata so I had somewhere to dive in if you need a hand.

Thank you!  I will be glad to help.  Suricata has not been started. I had been working on Snort until I took a break to work on the netmap stuff.  Send me a PM and we can swap e-mail addresses (if you don't have mine already).  Will be easier probably to correspond that way or by phone even.

Bill

Will do. I figured it would be best to attempt a smaller package, like cron, so I can get the process nailed down before moving onto something larger like Suricata. I'll be in touch once I get that done and we can come up with a plan of attack.

Offline doktornotor

  • Hero Member
  • *****
  • Posts: 8553
  • Karma: +956/-278
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: Working with Packages on 2.3 and Converting to Bootstrap
« Reply #5 on: December 04, 2015, 04:56:51 pm »
Guys... those packages that are adding their own user/group accounts - is that now safe to remove that code?
Do NOT PM for help!

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21388
  • Karma: +1432/-26
    • View Profile
Re: Working with Packages on 2.3 and Converting to Bootstrap
« Reply #6 on: December 04, 2015, 05:01:17 pm »
Such as?

If it's for a binary package, at least in theory, the pkg code should be handling that properly. I know PBIs had some issues there.  When in doubt, pkg add just the underlying program (not the pfSense pkg) and check for the user after.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline doktornotor

  • Hero Member
  • *****
  • Posts: 8553
  • Karma: +956/-278
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: Working with Packages on 2.3 and Converting to Bootstrap
« Reply #7 on: December 04, 2015, 05:26:00 pm »
Such as?

If it's for a binary package, at least in theory, the pkg code should be handling that properly. I know PBIs had some issues there.  When in doubt, pkg add just the underlying program (not the pfSense pkg) and check for the user after.

Well, the first one I hit was Avahi. Then, there's the infamous Squid thing with the ClamAV stuff where PBI was adding something with wrong UID/GID and the proxy user/group confusion/mess. Also, I recall there vaguely there was some issue related to the UID/GIDs handling with certain range being reserved for pfSense user manager and the accounts removed on reboot???
Do NOT PM for help!

Offline doktornotor

  • Hero Member
  • *****
  • Posts: 8553
  • Karma: +956/-278
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: Working with Packages on 2.3 and Converting to Bootstrap
« Reply #8 on: December 04, 2015, 06:52:17 pm »
Another note - I don't think this package makes any sense? Never had any GUI.

https://github.com/pfsense/FreeBSD-ports/tree/devel/net-mgmt/pfSense-pkg-iftop
Do NOT PM for help!

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21388
  • Karma: +1432/-26
    • View Profile
Re: Working with Packages on 2.3 and Converting to Bootstrap
« Reply #9 on: December 04, 2015, 06:57:13 pm »
Another note - I don't think this package makes any sense? Never had any GUI.

https://github.com/pfsense/FreeBSD-ports/tree/devel/net-mgmt/pfSense-pkg-iftop


It's just there so it gets built and available in the repo. It's console only and very handy to have available.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21388
  • Karma: +1432/-26
    • View Profile
Re: Working with Packages on 2.3 and Converting to Bootstrap
« Reply #10 on: December 04, 2015, 07:00:29 pm »
Such as?

If it's for a binary package, at least in theory, the pkg code should be handling that properly. I know PBIs had some issues there.  When in doubt, pkg add just the underlying program (not the pfSense pkg) and check for the user after.

Well, the first one I hit was Avahi. Then, there's the infamous Squid thing with the ClamAV stuff where PBI was adding something with wrong UID/GID and the proxy user/group confusion/mess. Also, I recall there vaguely there was some issue related to the UID/GIDs handling with certain range being reserved for pfSense user manager and the accounts removed on reboot???

None of the "official" users would collide with pfSense:

https://github.com/pfsense/FreeBSD-ports/blob/devel/UIDs
https://github.com/pfsense/FreeBSD-ports/blob/devel/GIDs

Packages in FreeBSD are supposed to specify their users in the Makefile, and the specifics of UID/GID/etc are pulled from the above files. For example:
https://github.com/pfsense/FreeBSD-ports/blob/devel/www/squid/Makefile#L36

I don't think we should have any code in the .inc files or elsewhere to manually make users any longer, if one isn't in the above list we should add it in and also tweak the pfSense package Makefile appropriately.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline doktornotor

  • Hero Member
  • *****
  • Posts: 8553
  • Karma: +956/-278
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: Working with Packages on 2.3 and Converting to Bootstrap
« Reply #11 on: December 04, 2015, 08:13:09 pm »
I don't think we should have any code in the .inc files or elsewhere to manually make users any longer, if one isn't in the above list we should add it in and also tweak the pfSense package Makefile appropriately.

OK, sounds good. Junk nuked. :)
Do NOT PM for help!

Offline jmurphyau

  • Newbie
  • *
  • Posts: 1
  • Karma: +0/-0
    • View Profile
    • jmurphyau
Re: Working with Packages on 2.3 and Converting to Bootstrap
« Reply #12 on: December 06, 2015, 08:05:06 am »
I had a bit of a look at this task and didn't notice any mention of anything relating to REST..

I noticed the pfSense blog post about converting to REST in 3.0 (https://blog.pfsense.org/?p=1588) - its quite old though.. Is this still the plan?

While in there changing the PHP pages it could be good if people were taking this into account as much as possible..

Something like:

  • Split functional code out into a [file]_rest.php that returns JSON
  • Change HTML page to make REST HTTP requests to execute their functions

Doing this now could get the process of converting to REST early (unless 3 is the next release directly after 2.3?).

I was taking a look at the sipproxd and it seems relatively small and thinking - depending on the time frame that packages need to be converted in - I might be able to take a look at converting that one in this manner? Could give a basic template to start moving to the REST structure..

I'm primary a web developer and work a lot with REST APIs (built in Django) and web front ends (the Ember JavaScript framework)

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21388
  • Karma: +1432/-26
    • View Profile
Re: Working with Packages on 2.3 and Converting to Bootstrap
« Reply #13 on: December 06, 2015, 08:12:09 am »
REST will be for 3.0 only and some time off (we haven't even began drafting the API) -- Seems like overkill to start all that now when we haven't decided on what we'll be using for any of that.

We'll be on 2.3 for a while, most likely there will even be a 2.4, that blog post is long-term goals.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline jdillard

  • Administrator
  • Sr. Member
  • *****
  • Posts: 534
  • Karma: +124/-1
  • Web Dude
    • View Profile
    • Personal Website
Re: Working with Packages on 2.3 and Converting to Bootstrap
« Reply #14 on: December 07, 2015, 02:26:32 pm »
I had a bit of a look at this task and didn't notice any mention of anything relating to REST..

I noticed the pfSense blog post about converting to REST in 3.0 (https://blog.pfsense.org/?p=1588) - its quite old though.. Is this still the plan?

While in there changing the PHP pages it could be good if people were taking this into account as much as possible..

Something like:

  • Split functional code out into a [file]_rest.php that returns JSON
  • Change HTML page to make REST HTTP requests to execute their functions

Doing this now could get the process of converting to REST early (unless 3 is the next release directly after 2.3?).

I was taking a look at the sipproxd and it seems relatively small and thinking - depending on the time frame that packages need to be converted in - I might be able to take a look at converting that one in this manner? Could give a basic template to start moving to the REST structure..

I'm primary a web developer and work a lot with REST APIs (built in Django) and web front ends (the Ember JavaScript framework)

I agree with Jimp, we should focus on getting 2.3 out to prevent what could be massive scope creep. Not saying it isn't important though. I was actually looking at make graph.php utilize an API over the weekend so that I could convert the traffic graphs to d3.j3 using JSON for the datasource. If you want to take a stab at converting various small data points to JSON APIs and need someone to bounce ideas off of let me know. I could probably squeeze a little bit of time in the week to hack on stuff like that.