Category Archives: xfdl

Army Forms Going to PDF…

After a year of no longer having support of XFDL/Lotus Form Filler documents it would appear that the Open XFDL project had moved on.  At least that was my thought but it has now become apparent that there is still a community in need of improving XFDL interfacing with databases.

Please add comments and discuss what is the future of the XFDL format.  Will this be a legacy format within the military or will it phase out soon?

 

XFDL with PHP 2…

Recent progress with doing XFDL forms in PHP.  I put some time into this project recently figuring that this web based viewer would be the most applicable.  Think if AKO’s MyForms, a setup where you can pass forms from one user to the next to get it filled out and signed; if that program never needed to leave the web?  With doing XFDL forms in PHP, that is possible!  My code is a mess right now so I will put up the PHP, Javascript for the page on a later post.

For now, go to the old version to see what the forms should be like and go to the new version to see progress on the viewer.

XFDL in PHP…

Lately, I have started working on the OpenXFDL project again, concentrating efforts in making a C++ application with WX Widgets that can compile for Windows, Mac, and Linux.  Getting frustrated with libxml2 and wanting to learn more of xpath queries, I decided to make a XFDL reader in PHP.  This can be applied easily as browser/e-mail plugins to view XFDL forms without the need of an external viewer.  I’ve been bad about posting but I’ll work to add more to this blog soon.  For now, here is a sight to try it out with.  Unlike previous stuff, you can do a straight XFDL form instead of converting to XML first.  Try it out!

GZip with ZLib

In an effort to use zlib in VBA, I had to first learn to use it.  I’m looking to gunzip the XFDL files to use the data as an MS  Office plugin.  In tinkering with zlib’s GZip functions, I found them to be very much like normal file handling in C.  Here’s a link to the official zlib reference manual, GZip is the tenth section.  So, to test it, I made a “micro-gunzip” which does work well.  When compiling, don’t forget to link to libz.a using gcc -o out -i in.c -lz.   Other than that, enjoy!  The basics are covered in the comments.

#include <stdio.h>
#include <zlib.h>
/* gcc filename.c -o outname -lz */

int main(int argc, char *argv[]) {
	// variables
	gzFile gFile;           // gzip file object
	FILE *pFile;            // out file object
	char *filen = argv[1];  // get the name of the file
	char buffer[100];       // buffer for reading

	// open files
	gFile = gzopen(filen, "rb");        // same as file
	pFile = fopen("example.out","wb");
	if(pFile == NULL || gFile == NULL)
		perror("Error opening file");
	else {
	    // write
		while(!gzeof(gFile)) {
			gzgets(gFile,buffer,100);   // file object first
			fputs(buffer,pFile);        // write to plain tex
		}
		// close
		fclose(pFile);
		gzclose(gFile);
	}
	return 0;
}

Microsoft Access : XFDL Viewer – Introduction

I know, I said it yesterday that it is rare that I develop on Windows, but this is a long promised application.  In 2007, my unit administrator (in the U.S. Army Reserves) suggested code that would allow batch loading XFDL forms from MS Access.  Due to the scope of the Apps 4 The Army project, I was limited to using a web application.  Now that the project is submitted and done, I am free to do my original plan, which is code to do the same process in MS Access.

Starting this today, I realized how much I hate doing Visual Basic.  Particularly, VBA is very painful!!  It’s not that the language is bad but it just always feels bulky and pieced together to me.  It seems to lack the professionalism of C/C++ and the flow of Python.  But, I may be alone in that.

I have a question for my readers though. Would anyone have an interest in seeing this project on SourceForge?  The Apps 4 the Army project is no longer my intellectual right, but a desktop application, a MS Office Plugin, etc… that’s all good to make public.  So let me know, if I get readers saying we’d like to help; then I will happily move this project to SourceForge!

DONE!!!

It is done!  The XFDL Loader is completed and it is functional.  It came down to the wire, with less than an hour left in submission time to complete but it did get done.  Here is a sample XFDL form and a sample CSV data file (RIGHT CLICK BOTH TO SAVE) to use.  The process to use the web application is (each step is a different page):

  1. Load the two sample files by file type.
  2. Set up data to merge to the form.
    • Use the drop down box on the right to select data by the header
    • Write data in the middle column input boxes to have default data for all forms.
  3. It’ll take awhile to generate the forms, be patient.  When completed, there will be a link to download a zipped archive of your forms.

And that’s it.  Be aware, because the project changed from PHP to Python, we had 10 days to generate the web application.  Therefore, there is no styling (it’s not pretty) and it might not work on all forms.  Also, it only selects text inputs.  Check boxes, more options, etc are yet to come, but this was needed to be functional by May 15th, and with only minutes to spare, it was done!

Approaching the final hours!

The project is due tomorrow!  I have a lot of work to do, but it finally appears possible to have this all done.  Thank you to my team member who contributed the code to correct the xml.dom.minidom’s output to be read correctly by PureEdge and Lotus Forms Viewer.  This will enable me to save, although I’m a bit sad that I spent so much time  this week working on that.  At this point, I need to complete the parsing scheme, develop a function to save the data to forms per the CSV file and then send back those files in a zipped archive.  All small steps, and assuming no road blocks, able to get done today.  A quick bit of documentation and packaging and this barely beta, functional web application will be ready for submission.  This may require an all-nighter and it’s down to the wire, but I think it will work.  Keep your eyes on the XFDL Loader as it will be updated through the day!

Frustration…

So I have now made a front page to upload a CSV file and an XFDL file for this XFDL web application. Nothing is more surprising than being a few short steps from having a working prototype to discover you’ve made a grave error. Here’s my mistake. I used python’s xml.dom.minidom module to parse and edit the file in it’s XML form. But when saving it back, it becomes obvious that the encoding causes some minor problems, but the major problem comes in the form for it’s saved state. Where the XFDL file will have a set of tabs, the minidom would correct these to read just which then is not able to be read by Lotus Forms Viewer or PureEdge! I’ll be frantically rewriting my parsing in other modules tonight to find one that is compatible in it’s write method!

XFDL Viewer Update…

As I dig through miles of XML, I realize I have not posted lately. I’ve been a bit busy meeting the May 15th deadline for my project. But here’s a quick update. First, don’t forget to view the update viewer. It’s not pretty, but it works to the point I’ve set it. I’ve changed from PHP to using Python CGI (a language I’m more comfortable with) and I’ve done what took two weeks in two days. Too bad this wasn’t thought of before!! Anyhow, I figured I’d pass along this bit of code from Neil Funk that shows a better method for decompression of an XFDL in Python:

#!/usr/bin/python
from base64 import *
import zlib

# OPEN FILES
inp = open('example.xfdl','rb').read()
out = open('example.xml,'wb')

# TAKE OUT THE MIME DATA
magic = inp.splitlines()[0]
data   = inp.split(magic)

# THE TRICKY PART -- Look to the python manual for explanation.
wbits = 15+32
newData = zlib.decompress(b64decode(data),wbits)

out.write(newData)
out.close()

And drumroll please, for Chris Hutton contributed this PHP code for recompressing an XFDL from XML.  I’m working to translate this to Python, but for the time being, an XML with an XFDL extension works.

// Export XML, gzip, and base64 encode. Append XFDL header information and output to user.
// XML is the raw data to be saved.
$newxml = $xml->asXML();
// chunk_split is new to me!
$output = chunk_split(base64_encode(gzencode($newxml)));
// Add the header!
$output = "application/vnd.xfdl;content-encoding=\"base64-gzip\"\n". $output;

header("Content-disposition: attachment; filename=\"{$filename}\"");
header("Content-type: application/vnd.xfdl");

echo $output;

So thank you all for your contributions!  I’ll try to keep this blog more up-to-date.

XFDL Viewer – Good and Bad.

I used to think of myself as a decent coder, but this XFDL project is a monster.  Particularly with the parsing schemes.  Let alone the May 15th suspense on the project!!  Oh well, if it doesn’t get done, I’m still working on it…  So I broke apart the XML and just to give a taste of the parsing for this, here’s my notes:

Mind you, I’m skipping the globals but here what needs to be parsed from there:

  • Title
  • Version
  • Fontinfo
  • Bgcolor
  • Print Settings
  • Bindings

This first block is the “field” form.  This is the main part to parse because this is where all values except checks and signatures will go.

<field sid="TO"> --> sid == div id
	<itemlocation>
		<ae>
			<ae>absolute</ae> -- style='position:absolute'
			<ae>y</ae>
			<ae>x</ae>
		</ae>
		<ae>
			<ae>extent</ae> -- dimensions
			<ae>l</ae>
			<ae>w</ae>
		</ae>
		<value>THIS IS WHAT NEEDS EDITTING!</value> - input, no border
		<broderwidth>0</broderwidth> --> style
		<fontinfo> --> style
			<ae>type</ae>
			<ae>size</ae>
			<ae>attribute</ae>
		</fontinfo>
		<justify>center</justify> --> style
		<scrollhoriz>wordwrap</scrollhoriz>
		<scrollvert>fixed</scrollvert>
		<next>TO</next> -> form taborder
		<previous>DATE_A</previous> -> form taborder
		<acclabel>lorum ipsum</acclabel>
		<format>
			<ae>string</ae> --> set form input type
			<ae>optional</ae> --> validation check
			<length>
				<ae>0</ae> vert
				<ae>18</ae> horiz
			</length>
		</format>
	</itemlocation>
</field>

Next part, check boxes.  Nothing too scary here.  Much the same as above.

<check sid=""> same
	<itemlocation></itemlocation> same
	<value>on|off</value>
	<fontinfo></fontinfo> same
	<next></next> same
	<previous></previous> same
	<acclabel></acclabel> same
</check>

Here we get complicated.  Nothing bad about the labels, except there form uses labels for definitions at the end and you’ll see it later where it is reused for a new purpose.  This could pose a good challenge!

<label sid=""> same
	<itemlocation></itemlocation> same
	<value>TEXT</value>
	<linespacing>1</linespacing> OPITONAL
	<fontinfo></fontinfo> same
	<fontcolor>black</fontcolor>
	<format></format> same
</label>

Lines, not too bad… will just be a simple parsing the xml to putting them on the page.

<line sid=""> same
	<itemlocation></itemlocation>same
</line>

Buttons!  These would be easy but they will associate with signatures.

<button sid=""> same
	<itemlocation></itemlocation> same
	<value compute="">FROM CERT</value>
	<type>signature|</type>
	<vfd_signmode>custom</vfd_signmode>
	<printvisible compute="">on|off</printvisible>
	<signformat>???WTF???</signformat>
	<signature>FOR BINDINGS</signature>
	<signer>FROM CERT</signer>
	<custom:onClick>function</custom:onClick>
	<signoptions>
		<ae>omit</ae>
		<ae>triggeritem</ae>
		<ae>coordinates</ae>
		<ae>ufv_settings</ae>
	</signoptions>
	<vfd_group>??</vfd_group>
	<format></format> same
	<previous></previous> same
	<next></next> same
	<image>FROM CERT?</image>
	<signatureimage>FROM CERT?</signatureimage>
	<signitemrefs>
		<ae>LOCK BINDS</ae> -- USE FOR VALIDATION
	</signitemrefs>
</button>

Not even touching signatures until I get the parsing scheme in place!

<signature>
	WILL DO LATER
</signature>

Data will be easy enough, just needs to be decoded and displayed.  Whew… may need to save the files to the server to display but I think HTML can handle an image from data.

<data sid=""> relates to signatures & images
	<filename></filename> -- image (optional)
	<mimedata encoding="base64-gzip"></mimedata>
</data>

Do I really need the toolbar if I’m not using it’s functions?

<toolbar sid="TOOLBAR">
	<bgcolor>
		<ae>gray60</ae>
	</bgcolor>
</toolbar>

Is IBM so dense they can’t find a new name for the toolbar tags instead of reusing label?

<label sid="TOP">
	<itemlocation>
		<ae>
			<ae>within</ae>
			<ae>TOOLBAR</ae>
		</ae>
		<ae>
			<ae>absolute</ae>
			<ae>0</ae>
			<ae>0</ae>
		</ae>
	</itemlocation>
	<image>PAGE1.ArmyLogoTop</image>
	<imagemode>clip</imagemode> -- style
	<active>off</active>
</label>

I won’t complain about this one!  Seems pretty simple, just like lines…

<spacer sid=""> same
	<itemlocation></itemlocation> same
</spacer>

So keep watching the blog… I’ll post more when I figure out how to handle all this data.  And this all needs to be cycled per page to output to the browser… yuck.

XFDL Viewer

The scope of the “Apps 4 the Army” competition that I am making my XFDL Viewer for is to use web apps.  So I did a quick change to PHP and read up and got a framework setup.  The idea is to be able to import the XFDL file and then parse the resulting XML to input values, either on a single form or by batch through an uploaded database.

The framework is complete and I’m pretty happy with the results as I’ve done much more application programming than web development, but PHP isn’t too bad.  After getting the frame work together, I started to work on parsing.  I mistakenly uploaded a XFDL file rather than the decompressed XML and I found something remarkable.  PHP’s SimpleXML apparently can decompress the XFDL file!! This takes out a very complicated step of my project!!!  I’m still working to figure out why the recompression does not produce the same output but that may not be a problem with the web interface.  Stay tuned for more updates.

XFDL in Linux :: Update

So, after talking with the Army’s G6 (essential the IT Department), the competition I’m entering (which because I’m not an active duty Reservist, I cannot receive prize money) is focused on web applications, not desktop applications.  The changes you can expect to see in my coding is that there will be quite a bit of php, perl and python that I’ll be using rather than the application side.  At the same time though, I’ll take what I do in this web application and use it for developing a desktop application and a Microsoft Office plugin that can be used to batch load files.  Just figured I’d acknowledge the reason for changes.  On the plus side of all of this, perl has a specific Mime module and php has a great and simple interface to xml parsing, so this may end up easier as a web application.

Python Curses – Custom Menu

Continuing on the same project as the previous post, I came to wanting to make a custom menu with curses in Python.  Realizing that there are functions to create menus in curses already, I wanted to build this fro m the bottom up.  The concept was to produce a menu that would highlight the selection change on the arrow keys or direct input, and then on a press of the Enter key the menu would return that selection.

Now, while writing this, you’ll see in my code that I took probable the least efficient way of building this menu, but it helps in making itself explanatory for the person learning curses.  As for the context of this coding, I built it as a function in my XFDL viewer, so there are 5 options in the menu.  I used win.keypad(0) to enable the use of the arrow keys but for some reason, the curses.KEY_UP was not being detected so the arrow key up and arrow key down are 259 and 258, respectively.  This does work though, I also have the menu catch numbers 1-5 and set the highlighted line accordingly.

def menu():
    curses.init_pair(1,curses.COLOR_RED, curses.COLOR_WHITE)
    screen.keypad(1)
    pos = 1
    x = None
    # I'm going to be lazy and save some typing here.
    h = curses.color_pair(1)
    n = curses.A_NORMAL
    while x != ord('\n'):
        # Gotta reset the screen from the root or lose the border, window, etc.
        screen.clear()
        screen.border(0)
        screen.addstr(2,2, "XFDL VIEWER", curses.A_STANDOUT)
        screen.addstr(4,2, "Please select an option...", curses.A_BOLD)
        # Detect what is highlighted by the 'pos' variable.
        if pos == 1:
            screen.addstr(5,4, "1 - XFDL -> XML",h)
        else:
            screen.addstr(5,4, "1 - XFDL -> XML",n)
        if pos == 2:
            screen.addstr(6,4, "2 - XML  -> XFDL",h)
        else:
            screen.addstr(6,4, "2 - XML  -> XFDL",n)
        if pos == 3:
            screen.addstr(7,4, "3 - Show XML",h)
        else:
            screen.addstr(7,4, "3 - Show XML",n)
        if pos == 4:
            screen.addstr(8,4, "4 - Exit",h)
        else:
            screen.addstr(8,4, "4 - Exit",n)
        if pos == 5:
            screen.addstr(9,4, "5 - DEBUG", h)
        else:
            screen.addstr(9,4, "5 - DEBUG", n)
        screen.refresh()
        x = screen.getch()
        # Is 'x' 1-5 or arrow up, arrow down?
        if x == ord('1'):
            pos = 1
        elif x == ord('2'):
            pos = 2
        elif x == ord('3'):
            pos = 3
        elif x == ord('4'):
            pos = 4
        elif x == ord('5'):
            pos = 5
        # It was a pain in the ass trying to get the arrows working.
        elif x == 258:
            if pos < 5:
                pos += 1
            else:
                pos = 1
        # Since the curses.KEY_* did not work, I used the raw return value.
        elif x == 259:
            if pos > 1:
                pos += -1
            else:
                pos = 5
        elif x != ord('\n'):
            curses.flash()
            # show_error() is my custom function for displaying a message:
            # show_error(str:message, int:line#, int:seconds_to_display)
            show_error('Invalid Key',11,1)

    return ord(str(pos))

I’ve highlighted the lines pertaining to my work around for the key pad.  This function will return the menu option and then that is processed for a reaction.  Reminder: the ‘screen’ object for my curses window is a global variable. I’m quite thrilled at the simplicity of this and the curses library, although I am disappointed in the lack of tutorials on the web deeper than typical ‘Hello World’ tutorials, but I hope these posts go to help others exploring this library!

Here’s a nice picture of the library in action:

Customize menu in action...
Be sure to comment on these tutorials and let me know if there is more detail needed or if they are helpful!!

Python Curses – Custom Progress Bar

Custom Progress BarIn making a visual editor for my XFDL project, I’ve decided to do the basic start in curses.  This is a module of python/library of C that I have not been able to utilize much at this point despite interest.  Unfortunately, there seems to be limiteexid documentation beyond the standard ‘Hello World’ type of program.  I’ll write this short tutorial with the assumption you can get through those tutorials and utilize this object.

The concept here is to create a progress bar in curses.  This example does not have any data but a basic understanding of python would allow you to create the refreshes on percentage of completion rather than time based as I have here.

def show_progress():
    #Create a window object.
    win = curses.newwin(3,32,14,10)
    # Add the Border
    win.border(0)
    # Current text: Progress
    win.addstr(1,1,"Progress ")
    # This is to move the progress bar per iteration.
    pos = 10
    # Random number I chose for demonstration.
    for i in range(15):
        # Add '.' for each iteration.
        win.addstr(1,pos,".")
        # Refresh or we'll never see it.
        win.refresh()
        # Here is where you can customize for data/percentage.
        time.sleep(0.05)
        # Need to move up or we'll just redraw the same cell!
        pos += 1
    # Current text: Progress ............... Done!
    win.addstr(1,26,"Done!")
    # Gotta show our changes.
    win.refresh()
    # Without this the bar fades too quickly for this example.
    time.sleep(0.5)

See, nothing to it.  The first line creates a new window object for and the next line adds its border.  I mapped out the relative positions for text, and you’ll want to customize this to what you need to use in your program, but for the sake of demonstration, it will show a period for each iteration.  As previously stated, you can customize this to be data or percentage driven in your program.  Don’t forget to call win.refresh() for each iteration you want to show on the screen!

XFDL in Linux :: Part 2

More for the ongoing series of producing an XFDL viewer in Linux.  In the previous tutorial, we decompressed an XFDL file, although I have had trouble recompressing the file.  It turns out that I need to do some experimentation and find the exact compression method used in gzip to be able to make the form readable.  That will be for the next update though.  I thought I would give a short preview of what’s next on this.

An XFDL file is an XML (xform) by IBM meant to run through their interpreter. IBM has some great documentation on this format.  PureEdge works much like a browser does to decompress the file by Mime-type and to then parse and read the file, including embedded binaries (for pictures, files, etc) and embedded coding (custom functions).  My interpreter will have a long ways to go so I’ll be happy to just be able to place my values in the correct fields.  I’m re-reading XML parsing within Python to make this an easy function, so be patient on that part.  But for those eager to see what I’m talking about, I’ve pasted a small section of XML from a decompressed XFDL.

      <field sid="NAME">
         <itemlocation>
            <ae>
               <ae>absolute</ae>
               <ae>9</ae>
               <ae>448</ae>
            </ae>
            <ae>
               <ae>extent</ae>
               <ae>461</ae>
               <ae>24</ae>
            </ae>
         </itemlocation>
         <value></value>
         <borderwidth>0</borderwidth>
         <scrollhoriz>wordwrap</scrollhoriz>
         <scrollvert>fixed</scrollvert>
         <fontinfo>
            <ae>Times New Roman</ae>
            <ae>10</ae>
            <ae>plain</ae>
         </fontinfo>
         <format>
            <ae>string</ae>
            <ae>optional</ae>
         </format>
         <previous>NAME</previous>
         <next>GRADE</next>
         <acclabel>d ay form 46 44-r, december 19 82.
ay p d. p e version 1.00.
edition of 1 august 19 77 is obsolete.
army reserve reenlistment data.
for use of this form, see ay r 1 40-1 11, the proponent agency is r c p ay c.
item 1. enter name using last name comma first name comma middle initial format.</acclabel>
      </field>
As you can see, there is a <value> tag for these nodes.  For my next post, I’ll write some python code to break this xml to an object that can print the label and insert a value into the xml.  There is a lot of work to interpret the embedded items, code and other tags, but this will be a start!

XFDL in Linux :: Part 1

Earlier, I wrote about using PureEdge Viewer, which is Windows software from IBM on Linux through Wine. This got me thinking, do we need to use Windows software. A quick look at the file and through Google and it’s easy to see that an *.xfdl file is a gzipped, base64 encoded xml file. So this is part one of what I hope to be a tutorial into designing software on Linux using python to open, read and write xfdl files in the same way as PureEdge Viewer. It gets annoying, to say the least, to need to open Windows in VirtualBox, or an actual install to read and edit *.xfdl files. The barriers I see at this preliminary point is to convert the xml to a readable image and then to make that editable where possible.

So, in this first part, we will do the very basic converting an *.xfdl file to an *.xml file. The code should be self explanatory but if there are questions, post them through comments and I’ll do my best at getting an answer to you.

#!/usr/bin/python
""" IMPORTS """
from base64 import *
import gzip, os, sys

""" DEATH!!! """
# Standard way to die...
def die(msg=None):
    if msg == None:
        msg = "Unknown error."
    print " [*] ERROR - %s" % msg
    sys.exit(1)

""" CHECK FOR FILE """
# No file name, then we have nothing to do!
if len(sys.argv) < 2:
    die("Did not specifiy a file name.")

""" GET FILE """
# In a more advanced version, this will check the magic value of the file as well to
# ensure it is an *.xfdl file.
filename = sys.argv[1]
print "Using %s" % filename

""" OPEN FILE AND SPLIT """
# Nothing tough, grab the magic number (1st line) and then store the rest as a variable.
f = open(filename,'r').read()
magic = f.splitlines()[0]
print "magic: %s" % magic
data = f.split(magic)[1]
print "Got Data."
del f

""" BASE64 DECODE """
# First we decode the base64.
f = open('temp.gz','wb')
f.write(b64decode(data))
f.close()
print "Base64 Decoded."

""" GUNZIP DATA """
# Yes, I know this writing to a file and then deleting it is ugly but I have not found
# a way to gunzip from a data stream.
f = gzip.open('temp.gz','rb')
gunzip_data = f.read()
f.close()
os.remove('temp.gz')
print "Gunzipped Data."

""" SAVE XML FILE """
# As this gets more advanced, it should be able to stay as a data stream for editing.
filename = filename.split(".")[0] + ".xml"
f = open(filename,'wb')
f.write(gunzip_data)
f.close()
print "Saved to temporary file '%s'" % filename

Nothing too involved here, simply open the file, strip out the first line and the decode the rest from base64 and gunzip that data to get the xml inside.  In the next tutorial, we’ll look at the structure of that xml, once I actually understand it or find decent documentation on it!