Tag Archives: army

Army Forms Going to PDF…

After a year of no longer having support of XFDL/Lotus Form Filler documents it would appear that the Open XFDL project had moved on.  At least that was my thought but it has now become apparent that there is still a community in need of improving XFDL interfacing with databases.

Please add comments and discuss what is the future of the XFDL format.  Will this be a legacy format within the military or will it phase out soon?

 

XFDL in PHP…

Lately, I have started working on the OpenXFDL project again, concentrating efforts in making a C++ application with WX Widgets that can compile for Windows, Mac, and Linux.  Getting frustrated with libxml2 and wanting to learn more of xpath queries, I decided to make a XFDL reader in PHP.  This can be applied easily as browser/e-mail plugins to view XFDL forms without the need of an external viewer.  I’ve been bad about posting but I’ll work to add more to this blog soon.  For now, here is a sight to try it out with.  Unlike previous stuff, you can do a straight XFDL form instead of converting to XML first.  Try it out!

I’m Back…

Sorry for anyone who watches this site.  After the last server upgrade, this site went offline and I was not able to focus my attention on it until I’d returned from my Army school.  Now that I’m back though, I have several projects getting ready to go, here’s a list of what to expect over the next couple months:

  1. Return to the garmin wardriver project.
  2. A python based minesweeper (ncurses).
  3. XFDL projects (primarily MS Access integration).

That’s just a short list of what is on my mind.  It’s great to be back on-line!

Slow Down…

Some of the avid readers may of noticed a slow down in the frequency of my posts.  That is because I am preparing to go to a school with the Army (BOLC) for the next four months.  Due to the pace of this school, I will be doing far less development and it’s likely I’ll have less internet access.  I have not gone away and this page is not dead though.  I will post as able while gone but everything will return to normal when I’m back home.

NCOER Spreadsheet – Done

Okay, so I’ve updated my code and I’m ready to release to the world my NCOER speardsheet.  Here’s a quick tutorial on how to use it.

  1. Download the Excel 2007 or Excel 2003 version. (Right Click and Save As)
  2. Ensure that you have enabled macros [ Office 2007 ] or [ Office 2003 ] for your MS Excel.
  3. Open the file you downloaded.
  4. Enter the unit you need the report for in the textbox on the left.
  5. Click Update
  6. It will prompt you for your AKO Username/Password to access the data.
  7. Wait a short time and it will automatically load and filter.
  8. Print or export the data as needed.

And that’s it.  I know military types read this blog (proven through the Open XFDL project) so please comment and let me know your opinion!

NCOER Spreadsheet

Last week, I complained about how bulky VBA (Visual Basic for Applications) can be.  My current project is to create a spreadsheet to keep track of upcoming NCOERs.  This is an easy task in anything but the most used and abused application the admin personnel in the Army use, Microsoft Excel.  Now, the data is easy enough to pull once you get an SSL connection with this website (to access you need to use an AKO Username/Password.)  The Interactive Web Response System will allow you to pull data on any past, due or current NCOERS (Evaluations).  This is great as I can track our evaluations as they move up the chain and will see when they are late.  Unfortunately, the web designers have only sorted the information by last name.

So, my basic program is complete.  I can make a connection (fortunately, MSXML6.0 makes an easy post request to an SSL website… something that if using Python, is not an easy task.)  It would appear that it is utilizing Internet Explorer’s library for the connection.  Once the connection is made and the data is loaded, then I parse.  Here is where VBA gets bulky.  Fortunately, I can use a reference to VBScript’s RegEx object, but it is far from being as complete as say, any other regex engine in any other language.  On top of that, parsing text through splits and such is an amazing pain.  Here’s an example.

Python:

import re
test = 'A simple "test of the languages" will show how bad VBA is!'
a = re.findall(re.compile(r'".*"',re.I),test)
print "Result = %s" % a[0]
## Result = "test of the languages"

VBA:

Private Sub CommandButton1_Click()
    ' must include Microsoft VBScript Regular Expressions 5.5 in references
    Dim re As New RegExp
    Dim testing As Variant
    Dim test As String
    ' to escape quotes "" can be used or chr(34).
    ' unbelievably, chr(34) is easier to read than ""
    test = "A simple " & Chr(34) & "test of the languages" & Chr(34) & " will show how bad VBA is!"
    re.Pattern = Chr(34) & ".*" & Chr(34)
    re.IgnoreCase = True
    Set testing = re.Execute(test)
    Range("A1").Value = "Result = " & testing(0)
    ' Result = "test of the languages" in "A1"
End Sub

See the difference?  I know it’s pretty close but keep in mind, this is a single line of text.  Multiple lines, large text files, etc. can be horrendous.  What in the world is left, mid and right anyhow?  I know 90% of this is that I refuse to touch this language if I can avoid it but still, this is ridiculous!!

Microsoft Access : XFDL Viewer – Introduction

I know, I said it yesterday that it is rare that I develop on Windows, but this is a long promised application.  In 2007, my unit administrator (in the U.S. Army Reserves) suggested code that would allow batch loading XFDL forms from MS Access.  Due to the scope of the Apps 4 The Army project, I was limited to using a web application.  Now that the project is submitted and done, I am free to do my original plan, which is code to do the same process in MS Access.

Starting this today, I realized how much I hate doing Visual Basic.  Particularly, VBA is very painful!!  It’s not that the language is bad but it just always feels bulky and pieced together to me.  It seems to lack the professionalism of C/C++ and the flow of Python.  But, I may be alone in that.

I have a question for my readers though. Would anyone have an interest in seeing this project on SourceForge?  The Apps 4 the Army project is no longer my intellectual right, but a desktop application, a MS Office Plugin, etc… that’s all good to make public.  So let me know, if I get readers saying we’d like to help; then I will happily move this project to SourceForge!

DONE!!!

It is done!  The XFDL Loader is completed and it is functional.  It came down to the wire, with less than an hour left in submission time to complete but it did get done.  Here is a sample XFDL form and a sample CSV data file (RIGHT CLICK BOTH TO SAVE) to use.  The process to use the web application is (each step is a different page):

  1. Load the two sample files by file type.
  2. Set up data to merge to the form.
    • Use the drop down box on the right to select data by the header
    • Write data in the middle column input boxes to have default data for all forms.
  3. It’ll take awhile to generate the forms, be patient.  When completed, there will be a link to download a zipped archive of your forms.

And that’s it.  Be aware, because the project changed from PHP to Python, we had 10 days to generate the web application.  Therefore, there is no styling (it’s not pretty) and it might not work on all forms.  Also, it only selects text inputs.  Check boxes, more options, etc are yet to come, but this was needed to be functional by May 15th, and with only minutes to spare, it was done!

Approaching the final hours!

The project is due tomorrow!  I have a lot of work to do, but it finally appears possible to have this all done.  Thank you to my team member who contributed the code to correct the xml.dom.minidom’s output to be read correctly by PureEdge and Lotus Forms Viewer.  This will enable me to save, although I’m a bit sad that I spent so much time  this week working on that.  At this point, I need to complete the parsing scheme, develop a function to save the data to forms per the CSV file and then send back those files in a zipped archive.  All small steps, and assuming no road blocks, able to get done today.  A quick bit of documentation and packaging and this barely beta, functional web application will be ready for submission.  This may require an all-nighter and it’s down to the wire, but I think it will work.  Keep your eyes on the XFDL Loader as it will be updated through the day!

Frustration…

So I have now made a front page to upload a CSV file and an XFDL file for this XFDL web application. Nothing is more surprising than being a few short steps from having a working prototype to discover you’ve made a grave error. Here’s my mistake. I used python’s xml.dom.minidom module to parse and edit the file in it’s XML form. But when saving it back, it becomes obvious that the encoding causes some minor problems, but the major problem comes in the form for it’s saved state. Where the XFDL file will have a set of tabs, the minidom would correct these to read just which then is not able to be read by Lotus Forms Viewer or PureEdge! I’ll be frantically rewriting my parsing in other modules tonight to find one that is compatible in it’s write method!

XFDL Viewer Update…

As I dig through miles of XML, I realize I have not posted lately. I’ve been a bit busy meeting the May 15th deadline for my project. But here’s a quick update. First, don’t forget to view the update viewer. It’s not pretty, but it works to the point I’ve set it. I’ve changed from PHP to using Python CGI (a language I’m more comfortable with) and I’ve done what took two weeks in two days. Too bad this wasn’t thought of before!! Anyhow, I figured I’d pass along this bit of code from Neil Funk that shows a better method for decompression of an XFDL in Python:

#!/usr/bin/python
from base64 import *
import zlib

# OPEN FILES
inp = open('example.xfdl','rb').read()
out = open('example.xml,'wb')

# TAKE OUT THE MIME DATA
magic = inp.splitlines()[0]
data   = inp.split(magic)

# THE TRICKY PART -- Look to the python manual for explanation.
wbits = 15+32
newData = zlib.decompress(b64decode(data),wbits)

out.write(newData)
out.close()

And drumroll please, for Chris Hutton contributed this PHP code for recompressing an XFDL from XML.  I’m working to translate this to Python, but for the time being, an XML with an XFDL extension works.

// Export XML, gzip, and base64 encode. Append XFDL header information and output to user.
// XML is the raw data to be saved.
$newxml = $xml->asXML();
// chunk_split is new to me!
$output = chunk_split(base64_encode(gzencode($newxml)));
// Add the header!
$output = "application/vnd.xfdl;content-encoding=\"base64-gzip\"\n". $output;

header("Content-disposition: attachment; filename=\"{$filename}\"");
header("Content-type: application/vnd.xfdl");

echo $output;

So thank you all for your contributions!  I’ll try to keep this blog more up-to-date.

XFDL in Linux :: Part 2

More for the ongoing series of producing an XFDL viewer in Linux.  In the previous tutorial, we decompressed an XFDL file, although I have had trouble recompressing the file.  It turns out that I need to do some experimentation and find the exact compression method used in gzip to be able to make the form readable.  That will be for the next update though.  I thought I would give a short preview of what’s next on this.

An XFDL file is an XML (xform) by IBM meant to run through their interpreter. IBM has some great documentation on this format.  PureEdge works much like a browser does to decompress the file by Mime-type and to then parse and read the file, including embedded binaries (for pictures, files, etc) and embedded coding (custom functions).  My interpreter will have a long ways to go so I’ll be happy to just be able to place my values in the correct fields.  I’m re-reading XML parsing within Python to make this an easy function, so be patient on that part.  But for those eager to see what I’m talking about, I’ve pasted a small section of XML from a decompressed XFDL.

      <field sid="NAME">
         <itemlocation>
            <ae>
               <ae>absolute</ae>
               <ae>9</ae>
               <ae>448</ae>
            </ae>
            <ae>
               <ae>extent</ae>
               <ae>461</ae>
               <ae>24</ae>
            </ae>
         </itemlocation>
         <value></value>
         <borderwidth>0</borderwidth>
         <scrollhoriz>wordwrap</scrollhoriz>
         <scrollvert>fixed</scrollvert>
         <fontinfo>
            <ae>Times New Roman</ae>
            <ae>10</ae>
            <ae>plain</ae>
         </fontinfo>
         <format>
            <ae>string</ae>
            <ae>optional</ae>
         </format>
         <previous>NAME</previous>
         <next>GRADE</next>
         <acclabel>d ay form 46 44-r, december 19 82.
ay p d. p e version 1.00.
edition of 1 august 19 77 is obsolete.
army reserve reenlistment data.
for use of this form, see ay r 1 40-1 11, the proponent agency is r c p ay c.
item 1. enter name using last name comma first name comma middle initial format.</acclabel>
      </field>
As you can see, there is a <value> tag for these nodes.  For my next post, I’ll write some python code to break this xml to an object that can print the label and insert a value into the xml.  There is a lot of work to interpret the embedded items, code and other tags, but this will be a start!

XFDL in Linux :: Part 1

Earlier, I wrote about using PureEdge Viewer, which is Windows software from IBM on Linux through Wine. This got me thinking, do we need to use Windows software. A quick look at the file and through Google and it’s easy to see that an *.xfdl file is a gzipped, base64 encoded xml file. So this is part one of what I hope to be a tutorial into designing software on Linux using python to open, read and write xfdl files in the same way as PureEdge Viewer. It gets annoying, to say the least, to need to open Windows in VirtualBox, or an actual install to read and edit *.xfdl files. The barriers I see at this preliminary point is to convert the xml to a readable image and then to make that editable where possible.

So, in this first part, we will do the very basic converting an *.xfdl file to an *.xml file. The code should be self explanatory but if there are questions, post them through comments and I’ll do my best at getting an answer to you.

#!/usr/bin/python
""" IMPORTS """
from base64 import *
import gzip, os, sys

""" DEATH!!! """
# Standard way to die...
def die(msg=None):
    if msg == None:
        msg = "Unknown error."
    print " [*] ERROR - %s" % msg
    sys.exit(1)

""" CHECK FOR FILE """
# No file name, then we have nothing to do!
if len(sys.argv) < 2:
    die("Did not specifiy a file name.")

""" GET FILE """
# In a more advanced version, this will check the magic value of the file as well to
# ensure it is an *.xfdl file.
filename = sys.argv[1]
print "Using %s" % filename

""" OPEN FILE AND SPLIT """
# Nothing tough, grab the magic number (1st line) and then store the rest as a variable.
f = open(filename,'r').read()
magic = f.splitlines()[0]
print "magic: %s" % magic
data = f.split(magic)[1]
print "Got Data."
del f

""" BASE64 DECODE """
# First we decode the base64.
f = open('temp.gz','wb')
f.write(b64decode(data))
f.close()
print "Base64 Decoded."

""" GUNZIP DATA """
# Yes, I know this writing to a file and then deleting it is ugly but I have not found
# a way to gunzip from a data stream.
f = gzip.open('temp.gz','rb')
gunzip_data = f.read()
f.close()
os.remove('temp.gz')
print "Gunzipped Data."

""" SAVE XML FILE """
# As this gets more advanced, it should be able to stay as a data stream for editing.
filename = filename.split(".")[0] + ".xml"
f = open(filename,'wb')
f.write(gunzip_data)
f.close()
print "Saved to temporary file '%s'" % filename

Nothing too involved here, simply open the file, strip out the first line and the decode the rest from base64 and gunzip that data to get the xml inside.  In the next tutorial, we’ll look at the structure of that xml, once I actually understand it or find decent documentation on it!

VirtualBox – USB Devices

As a military member, I am issued a CAC card, which is a smart card carrying a PKI certificate for logging onto web interfaces and signing documents.  As  Linux user, I’ve been frustrated by the Windows specifics for utilizing CAC cards and have been happy to find alternatives through libcoolkey and pcsc_lite.  These work well with Firefox and I can log onto web interfaces just fine, but have not had a viable, Linux solution to signing documents.  In the military, we use an IBM product, PureEdge to utilize our forms which are *.xfdl documents.

The solution I have come to, although not purely Linux, allows me to sign documents without rebooting to another OS.  For testing purposes, I have had Windows XP running in VirtualBox so I decided to utilize that.  The problem came in that I was using virtualbox-ose and needed to not use the open source version for access to the USB devices.   The process then is relatively and well documented with simple Google searches:

  1. Set vboxuser in the user’s group.
  2. Configure your USB devices (in this case the CAC card reader) to be detected by the guest OS.
  3. Run the guest OS and install ActivClient 6.0 and Silinas ApproveIt to be able to sign *.xfdl documents.

See, simple… hmm, that sounds too close to Windows 7 add but I promise it is not!  I may be a PC but Windows 7 was NOT my idea!!